You already dealt with vision observations to implement the rectangle detection. Now you need to process the observations coming from your Inceptionv3 machine learning model. VNClassificationObservation is a powerful class that delivers a standardized way to access both the confidence and the identifier—this could be a specific object that was detected—of a predicted observation.
- [Instructor] We have already achieved a lot. With our VisionML application, we can already detect rectangles live from the camera, we are doing on device processing which is really cool, and now we have added our Machine Learning Model which detects the dominant objects present in an image from a set of 1000 categories such as trees, animals, and so on. And we have already created our request for the classification here with the VNCoreMLRequest using our CoreMLModel Inception 3 and now all that's left to do for us is actually handle this classification using the observations that we get and to do that, I'm going to continue writing some code here in our handleClassification function in line 78 in our ViewController.swift file and I'm starting with a guard statement creating a new object called observations.
And we're going to find these observations in our request and its results error and if this fields, then what we should do is print our error first, let's say no results and let's also append the error and it's localizedDescription and we should also return from this function because without the observations, we cannot really continue. And now that we have these observations, we need to do three things or we can do three things.
First of all, I'd like to cast them to classification observation objects because at the moment they're not. And in these classification observation objects, we get for example, a confidence property and an identifier and this is going to be really cool so we're going to use three sequence operations which is flatMap, filter, and map to achieve that. So let me first of all create a new classifications area and I'm going to use my observations area and just accessing the elements from zero to four.
And then I'm performing first of all the flatMap operation, which allows me to use an object here using dollar sign zero and cast that to a VNClassificationObservation object. So at the moment, after writing this line of code, I only have objects now of VNClassificationObservation in my classification area. What I can do now is also filter this area and what I'd like to do here is just using objects that do have a confidence, a vision confidence of more than 30% and this is the level of confidence normalized to a value between zero and one where one is the most confident.
And I'd like to filter out everything that is below 30%. Depending on your Machine Learning Model, you have to play around with this value to get the results that you actually want and then the last thing I'd like to do is use the map function because now I just want to have my identifiers in this classifications area. So I'm again using dollar sign zero accessing the identifier object which is a string and this is what I was talking about in the last video.
Normally when we're working with Machine Learning Models, we have to use the properties that are defined under outputs here like classLabelProbs and classLabel but in this case we're using vision and we're using these VNClassificationObservation objects. Every time you use these objects, you get this confidence and identifier property whatever Machine Learning Model you're actually using. And now, the last thing I'd like to do is going through classifications on my area so using a classification object in my classifications area and for each element that I find here, I'd like to add that to my text view.
So I'm using DispatchQueue.main.async because this is going to be performed on the main thread, UI updates, always need to happen on the main thread using self.objectTextView and its text and add my classification and this is actually it. Let's quickly build that, see if it works. And indeed it does, we get one minor warning here and we are just going to fix this using a String(describing) to silence this warning and then we can build that again.
And if we now run this on our device, and point it to a few objects, then you can that this beautifully tags almost every object perfectly and immediately because we're doing everything live on device without having to send anything to a third-party server or using a third-party library.
- What are machine learning, Core ML, Vision, and NLP?
- Adding a machine learning model to a project
- Getting predictions from machine learning models
- Converting existing machine learning models for Core ML
- Classifying images and detecting objects with Vision and Core ML
- Analyzing natural language text with NSLinguisticTagger