Deploy and optimize

- [Instructor] As an organization, you need to recognize that there's never a final version of an AI model. Your AI model work is never done. It would need to be updated and improved over time. This means that the data scientist still need to review and monitor work on previous projects even after the models are on production. This is to ensure that the models stay accurate as defined by the business. One of the things you and your team need to be thinking about is how to monitor the models. How can you make sure that your model is healthy? Are the models degrading? Let's go back to the e-commerce example. There are going to be new products that will need to be added to the store. Some of the items in the store might be seasonal and the sales reflect that. New techniques or algorithms might be available that data scientists might want to try out and see if they can get better performances with. Many applications have a dashboard to be able to monitor the health, CPU usage, and response times. Why not have a dashboard for your AI models? If you have the e-commerce store, you can track spikes or drops in some of the metrics you are using. Was there any underlying issue with some of the data? Can you take the recent data set and pass it through a development model and replicate the model behavior? Perhaps the model that we've created requires certain input features, but for some reason, those sets of inputs are missing in production. How does the model behave in this instance? So how often should a model be retrained? It depends on what they're predicting. There are always unforeseen external changes that can affect the accuracy of machine learning models. Seasonal changes and weather would have an impact on purchases of an online retailer. Marketing campaigns can affect sales. Here's an example. We are showing the model performance over time. The darker the shade of red in the heat map, the lower the model performance. The darker the shade of blue, the better the model performance. Another consideration is that the model that works for one geographical location might not work for another. Demographics, weather, and tastes vary across geography, and need to be taken into consideration for models to work well. Now, let's say your core business was around streaming sports programs globally. And you want it to provide recommendations to customers for what they should watch. Then geography is hugely important. Soccer, as it's known in the US, or football, as it's called everywhere else, is hugely popular in Europe, South America, and parts of Asia. The Super Bowl is typically popular mainly in the US, and cricket is very popular in South Asia. Things are complicated by say the football World Cup that comes around every four years. Those who don't normally watch football are drawn to support their national teams. The AI model needs to be able to take into account both geography and timing of events, and as you can imagine, this is not trivial. I was working on a health project where we were trying to predict if patients would require unplanned emergency care. So that's things like patients turning up at an accident and emergency unit or requiring an ambulance. And as you can imagine, demographic, geography, and past health can play a significant role. This is further complicated in cities where different parts of the population might live in certain geographies and a model developed for one hospital in one location might not apply or work in another part of the country. So how often do we need to update the models? If you have a deep-learning application using say face recognition, you would not expect there to be significant changes over the months. Many banks in the UK are starting to use voice recognition as a means of authentication. Now, I'm quite interested to find out how soon they prompt me to re-record my voice as part of its model training. Similarly, you would not expect significant changes for natural language models. There might be new vocabulary added, but this is over several months or years. Economic models need to change a little more frequently. For a country, this might be based on a monthly or quarterly basis when growth results are released by economists or the government. Fraud and spend analytic models need to be updated fairly regularly to capture the latest means of fraud. And finally, any algorithmic trading system needs to be able to deal with real-time changes and information that it's getting from buyers, changes in the economic outlook, news updates and so on. So frequent changes would need to be made for such a model. This means that real-time AI models might require dedicated teams working on them to update daily changes. Fraud and spend analytic teams need to be tracking and updating models frequently to capture any new techniques or methods. Fraudulent activity makes up a very small percentage which adds to the challenge of reducing the number of false positives. Economic models typically require monthly changes and will probably not require a dedicated team. Similarly, monthly changes to image-based deep learning models is adequate. The key thing here is to track the models and to be able to determine when there is model degradation and to deal with it. What are some of the signs that indicate you need to retrain your AI model? For mobile and telecom providers, one of the biggest challenges they face is customer churn. Often, telecom providers will profile their customers, so those who are loyal, those who are likely to leave, which are the most valuable customers based on their spend or which package they're on, and now, normally, the highest churn are from the new customers who've been around for less than six months. So if you plot these different profiles out and find that the distribution seems to change from the training data to real-world data, then it's probably an indication that you need to review the model. Model retraining normally begins with reviewing and understanding the data that caused the model to behave unexpectedly.