Join Doug Rose for an in-depth discussion in this video Ignore business value, part of Learning Data Science: Using Agile Methodology.
- Even when your team finds something interesting, you still have to connect it to real business value. It's not easy to connect exploration to business value. Often in data science, you don't know the business value, until after you've found the insight. You have to go through the entire life cycle, before you can deliver anything interesting. That's one of the key benefits of working in sprints. You'll deliver stories a little bit at a time, every two weeks. Each sprint, you'll build on what you know. The research lead can evaluate your insights and connect it to real business value.
If the team is on the wrong path, then they can pivot to something more interesting. I once worked for a retailer, that was trying to improve their worker safety. They created a Hadoop cluster, that collected all of their unstructured data. The cluster had video, images, and injury reports. The data science team used this data to create a word cloud of all the organization's job injuries. Then the team presented the cloud at their storytelling session. The team had a simple visualization and started to tell their story.
You could see everyone in the room rub their hands or cross their legs, as the team went through common injuries. At the end of the session, the data analysts said that they would use the next sprint to refine their analysis. They would create data visualizations that told a deeper story. It would have more specific injuries. A stakeholder in the room asked the team a simple question, why were they focused on the injuries and not on the equipment that caused the injuries? Everyone in the room could empathize with the people who were injured.
Still, the real value is in trying to prevent future injuries. The team could use predictive analytics to tell if the equipment is too dangerous. The data science team had been focused on who was injured. They were difficult stories and needed to be told. Still, the real business value was trying to prevent future injuries. The team needed to pivot and look at the equipment they were using. That was an entirely new data set to explore. If the team hadn't been working in sprints, then they would have spent months, or even longer, refining and exploring the injuries.
They would tell stories, but not necessarily the ones stakeholders wanted to hear. Instead, the next sprint, the team started focusing on dangerous equipment. They built on their previous data and told a whole new story about dangerous equipment and processes. It's not unusual for data science teams to explore the data without a clear connection to business value. In fact, the Gartner Group estimates that 85% of data analytics teams work on data with no clear connection to business value.
Some of this is due to the nature of the work. It's difficult to evaluate what you don't know. The other part is making sure that you have a clear connection to the stakeholders. Your research lead will work with the business to connect the team's insights to real value. Working in sprints allows the team to quickly pivot when the team finds something interesting. The stakeholders might not always know where to find business value. Instead, they're much more likely to know where not to go. Still, that feedback loop is essential to keeping the team on track.
Knowing where not to go, may eventually lead you on the right path. The data science team should be doing interesting work. It's one of the places in the organization, where you can build up real insights. Yet the team won't be immune to typical business pressures. If your data science team isn't producing real value, then it won't be long before stakeholders start to question your work. Most data science teams work much differently than the rest of the organization. If you don't quickly show business value, then it's unlikely that you'll be around long enough to make a difference.
The best way to create value is by having a tight feedback loop between the business and your team. The stakeholders should know what the team is working on every sprint. That work should be clearly connected to something that they see as valuable. Each storytelling session, try to tell a simple story of what the team learned and how it helped the rest of the organization. These sessions are essential to keeping the team working and focused on valuable insights.
This course shows how to structure your work within a two-week sprint. See how to work within a data science life cycle (DSLC)—a methodology for cycling through questions, research, and reporting every two weeks. Explore key practices to help your team break down the work so it fits within a two-week sprint. Learn how to use tools like question boards to encourage discussion and find essential questions. And most importantly, learn how to grow your team's shared knowledge and avoid common pitfalls.
- Defining data science success
- Determining project challenges and criteria for success
- Using a DSLC
- Iterating through DSLC sprints
- Creating a question board
- Breaking down your work
- Adding to organizational knowledge
- Avoiding pitfalls