Join Jonathan Fernandes for an in-depth discussion in this video Setting binary model's predictive performance, part of AWS Machine Learning by Example.
- [Instructor] So let's take a look at our banking scenario and head back to the AWS console. And let's look at the summary section under the evaluations. So we see here that we have an AUC score of 0.825. But remember, we said that the closer that figure is to one, the better the model's quality score. So let's explore the performance. And we can see that our score threshold is 0.5. Now, you can control that cutoff for what the model considers a positive prediction by increasing the score threshold until it considers only the predictions with the highest likelihood.
So remember that in this instance, each false positive costs the campaign money, because remember, false positive is when we predict someone will purchase the product, but they end up not doing so. So as a business, you might want to focus the campaign only on those who will subscribe to the product. And perhaps you might only want to focus on the top 5%. So I can do that by moving this cutoff. And you can see that as I move this across, the number of false positives decreases, and I might then want to just focus on the top 5%.
So you can see that it's currently at 9%, and I move it across until all of the records are predicted as one, and I've got that at 5%, and you can see that my threshold score is 0.84. So you can see that as we move this threshold vertical line across, the number of false positives decrease. So now, instead of our machine learning model predicting customers with a score of greater than 0.5 as being a one, and less than 0.5 as being a zero, we have this higher threshold, where we're targeting the 5% of the customers that we know will purchase this product.
So this means that only those greater than 0.84 will be predicted as a one, and those less than 0.84 will be a zero. So what you want to do is then save that score threshold at 0.84, and that means now that anything over that score threshold of 0.84 will be predicted as a one, and anything less than that will be a zero. So we're happy with 5% of the records being predicted as one, and this is achieved with a threshold of 0.84.
So in the next video, we will use the machine learning model to generate predictions.
- Learning algorithms and hyperparameters
- Preparing data for AWS
- Using binary, multiclass, and regression techniques
- Creating a datasource
- Generating predictions
- Creating and interpreting batch predictions
- Additional AWS capabilities