Learn how to judge the accuracy of a recommendation using using precision and recall.
- [Narrator] How do we actually know that the movie ratings we predict with our recommendations system are correct? To measure the accuracy of our predictions, we're going to use a standard statistics metric called root-mean-square-error or RMSE. RMSE is a measurement of the difference between the user's real movie ratings and the ratings we predicted for the same movies. The lower the RMSE, the more accurate the model. An RMSE of zero means our model perfectly guesses user ratings. An RMSE of one, means we are off by about one star on average when predicting user ratings.
When measuring the accuracy of our recommendation system, we need to make sure the data we are using to validate the system is data the system has never seen before. Otherwise, it's not a fair test. So, we'll randomly split our movie ratings data into two groups, the first 70% of data will be our training data set. We'll use the training data set to do matrix factorization and to create the U and M matrices. The other 30% of data we'll hold back as the testing data set. We'll use the testing data set to check how well the system makes predictions against data it's never seen before.
Let's take a look at measure_accuracy.py I've already split the data set into two different files movie_ratings_data_set_training.csv has 70% of the reviews and movie_ratings_data_set_testing.csv has the other 30%. We'll load both files into separate data frames using the Panda's read CSV function. Next, we'll use Panda's pivot table function to create separate ratings matrices for the training data and the test data. Then, we'll perform matrix factorization on only the training data.
Then, we'll multiply U and M to get the predicted ratings. Now, we can calculate the error rates using the RMSE function and matrix factorization utilities. To use the function, we just need to pass in the two arrays that we want to compare. First, we'll compare the training data to the predicted ratings, we called as_matrix on the training data frame so it gets passed in as a num.py matrix data type. Then, let's also compare the test data to the predicted ratings, then finally we can print out both numbers. Let's run the code and see what kind of results we get.
Right-click, choose run, okay, we got a training RMSE of 0.24 and a testing RMSE of about 1.2. The low training RMSE shows that our basic algorithm is working, the testing RMSE is the more important number because it tells us how good our predictions are. A score of 1.2 means that our system's a bit more than one star off on average when predicting the user's ratings. One thing we can do to try to improve the system is to adjust the regularization amount parameter. However, it's a trade off, more regularization will raise the training score, but it may lower the testing score.
One limitation that we have in this example is that we only have a few hundred movie reviews to work with, the best thing we could do to improve accuracy in this case, is to get more user reviews. More movie reviews will give our system more information to work with so it can do a better job of making recommendations. When building your own recommendations systems RMSE is a helpful tool to see how well the system is working, but RMSE isn't the final word. The ultimate test of a recommendation system is how much your users like the recommendations they get. So, don't worry about making tiny improvements in RMSE if it doesn't actually make the user experience better.
Recommendation systems are a key part of almost every modern consumer website. The systems help drive customer interaction and sales by helping customers discover products and services they might not ever find themselves. The course uses the free, open source tools Python 3.5, pandas, and numpy. By the end of the course, you'll be equipped to use machine learning yourself to solve recommendation problems. What you learn can then be directly applied to your own projects.
- Building a machine learning system
- Training a machine learning system
- Refining the accuracy of the machine learning system
- Evaluating the recommendations received