Evaluating success in your AI and ML projects

- Ultimately, the success of your project … will be judged by the measures it improves. … So we need to talk through … how we're going to measure your project. … As with all technology projects, … you need to track two sets of metrics. … The first set let's you know … how well the technology itself is working … and whether people are actually using it as intended. … These are leading indicators. … Leading, because you only get benefits downstream … if the technology is working and being used. … Remember, for many projects we need training data … that includes the desired answer or output … as well as the input data. … That way you train the algorithm … both with the input and the output. … Well, that's true, but there's an important detail. … You don't train with all the data. … Some of it you set aside to use for testing. … As a general rule of thumb, … you'll use 70% of your data to train your algorithm, … leaving 30% to test. … Sometimes you can get by with a smaller test set. … Your data scientists can help you make that call. …