The benefits and challenges of the scrum methodology

- [Instructor] Here are a couple of reasons why I think the Scrum methodology is great when introducing an AI project into an organization. You see, it's predictable. You know exactly what will be done in the sprint, so you can predict what the output will be. I find it's adaptable, you can change things as you go along. As some of the product owners or stakeholder's priorities change after a week or two, you can change some of the priorities starting from the next sprint. And it's transparent, the teams have a chance at the daily Scrum meetings to communicate and work together. The review and demo meetings is an opportunity to showcase what the teams have achieved during a sprint cycle. Now, I have recommended the Scrum methodology because I've seen it work for AI projects. But you won't be surprised to find that there are some people who think that the Scrum methodology isn't the best option for an AI project. So here are some of the objections. The Scrum framework has gained a lot of popularity in software engineering, and at the end of each sprint, you expect it to deliver something or deliver incremental features. Now if you're working with an AI or data science project, a good chunk of your time at the start of the project is spent cleaning your data and getting it into the required format. After you're done with that, then there's a lot of exploratory data analysis that takes place. It doesn't look like you're delivering anything. Now, this is a fair objection, so how do you deal with it? Perhaps the data science team should provide a report of findings at the end of each week. They could try out several hypotheses on the data and show the results, or they could create visualizations of some of the data. Now for the data scientists, you might not be delivering the AI models right yet, but based on the exploratory analysis of the data, there are trends and insights that you can report on. Another criticism of using the Scrum methodology for AI projects is timeboxing. So timeboxing works really well for software projects where you can define exactly what functions or features you're going to develop. Data science and AI projects are experimental projects, and they might take a lot of time to yield any results. So to counter this, the data science team is to be ruthless in defining exactly what will be done. It's easy to go down a rabbit hole, but the focus needs to be on the deliverables for the week. Today, there are tools that can automate a good chunk of exploratory analysis, and significantly cut down the time that this would normally take. Being able to focus on deliverables and finding the balance between exploring data and creating deliverables also comes with experience.