This course focuses on the Apple machine learning frameworks Core ML, Vision, and NLP. To save some time, you're not holding yourself up with the AVFoundation code required to display live camera images on your iPhone screen. Still, get a quick overview about the app's boilerplate code that you can use to follow along.
- [Instructor] Before we get started implementing all of the features that you've already seen like the rectangle detention and the life object classification, I'd like to give you a quick overview about the extra project that I've already created for you. It is called Vision ML, and you will find it in your exercise files. And if you open up the main storyboard, you will find three elements here. We have just the label stating Object Classification. We're not going to do anything with that. But we have the text view that is going to display the classification results, and we have a special UI view, and I've sub-classed UI view actually to create a special class.
It is called preview view, that is going to let us draw the rectangles that we have detected directly on screen. And if we open up the preview view, you can see that we have just a mask layer here, which is an area of CA shape layers, and we're using this preview view also to store our video preview layer, and we're having a function here, which is called draw layer, and this draws the rectangle into our view, and we're setting a corner radius, choosing a nice background color, and so on.
So this is just to create the rectangle and to display it once we have found a rectangle using the vision framework. And what we also have here is an extension for our view controller, which also adopts the AV capture video data output sample buffer delegate. So this is a delegate that we are going to use later to trigger the image recognition or to trigger the V object classification here. We prepare everything to display the live camera feed.
We're using a capture session here. We're checking if we have available devices. We're beginning this session and so on, since we are dealing with Core ML and the vision framework here. We're not going into too much detail here. But important for all the vision framework later on is that we have a function that returns the acts of orientation from the device orientation. This is important for the object classification later, but we are just going to use this function, and you can also use this function in your own projects.
So this is the view controller and camera extension that we are going to use. And especially here, we are interested later in the capture output dit output sample buffer from connection function, because here, we are going to perform the detection, the rectangle detection, and also the object classification. And in our view controller, we have some outlets for the two elements in our storyboard for the preview view, and for the object text view, we have some properties here for the live camera.
This is all part of AV foundation here, AV capture session, AV capture device, and also in property to store the position of our camera, which is in our case the back. And in the view will appear function, we just call the prepare camera function that I've added in the view controller and camera extension. But don't worry about the AV foundation code. We're just focusing on the vision framework and Core ML here.
- What are machine learning, Core ML, Vision, and NLP?
- Adding a machine learning model to a project
- Getting predictions from machine learning models
- Converting existing machine learning models for Core ML
- Classifying images and detecting objects with Vision and Core ML
- Analyzing natural language text with NSLinguisticTagger