Join Doug Rose for an in-depth discussion in this video Work without objectives, part of Learning Data Science: Using Agile Methodology.
- Your data science team will want to use the DSLC so they can tell interesting stories each sprint. These practices will help your team explore the data and ask interesting questions. These practices alone should help get your team focused on exploring. Still, for many teams, the biggest challenge will be trying to change your organization's mindset. Most organizations still see work as a series of goals and objectives. That's why most key organizational roles focus on management and compliance. The typical project manager ensures that the team complies with the project plan.
A lead developer maintains certain coding practices. A quality assurance manager enforces standards like Six Sigma. Even the CEO is focused on setting clear goals for the rest of the organization. All these popular roles focus on compliance. They help ensure that the organization stays true to their objectives. The people in these roles tend to be very influential. Chances are, they want to apply this objective-driven mindset to your data science team.
That's a real challenge. Remember that data science is about exploration. That makes it difficult to work off of typical objectives. Exploration, by definition, is about looking for something unfamiliar. That's much different from objectives, that are about staying true to your intended purpose. You can certainly mix exploration and objectives. If you find yourself in a new city, you might have an objective to find something to eat. Then you'll explore the best places for good food.
You have the objective of finding dinner, but you're still open to exploring new ideas. The problem is that most organizations aren't quite this flexible. They tend to narrowly define their objectives. The objective itself becomes the focus. A team isn't celebrated for changing course after they find something new. Instead, a successful team should have a well-defined objective and meet their goals within an expected timeframe. This focus on objectives can create a very difficult environment for exploration.
Let's go back to our running shoe website. Imagine that your data science team is given the objective of creating a report which breaks down purchases by credit cards. While the team is exploring the data they notice something unexpected. It looks like there's a positive correlation between shoe sales and customer ratings. You might expect that a shoe with the highest rating would have higher sales. The data science team noticed that any rating leads to higher sales. The lowest selling shoes were the ones that had no rating at all.
Based on this data, the team pivots to take advantage of this new discovery. They create an entirely different set of reports that correlates ratings with top shoe purchases. At the storytelling session, they tell their story. They talk about how the customer is least likely to buy a shoe if they think it's unpopular. In fact, a shoe with a terrible rating is still more likely to sell than a shoe with no rating. This new discovery was completely unexpected. The team had the objective of looking at new credit card data and then pivoted to start looking at ratings data.
In a typical project, this would be unacceptable. You wouldn't want your teams to have a set objective and then change direction based on some new discovery. This kind of shift generally requires approval from management. For a data science team, this is perfectly acceptable and even encouraged. In fact, many data science teams try to define their objectives as broadly as possible. They might have the open-ended notion of looking for patterns. They may just explore the data to see if anything sticks out, to see if there's something interesting in the data.
These teams find that clearly defined objectives can often be an impediment to discovery. When you're on a data science team, try to remember that you're doing something different from the rest of the organization. Most teams in infrastructure, sales, human resources, and management have no trouble establishing clearly defined objectives. With data science, you can get the most value from your data if you focus on discovery. Many managers have spent years focusing on clearly defined objectives. A data science team is doing something new and that might not be so easily accepted.
You should work closely with your managers to communicate this difference. Don't underestimate the challenge of trying to change their expectations.
This course shows how to structure your work within a two-week sprint. See how to work within a data science life cycle (DSLC)—a methodology for cycling through questions, research, and reporting every two weeks. Explore key practices to help your team break down the work so it fits within a two-week sprint. Learn how to use tools like question boards to encourage discussion and find essential questions. And most importantly, learn how to grow your team's shared knowledge and avoid common pitfalls.
- Defining data science success
- Determining project challenges and criteria for success
- Using a DSLC
- Iterating through DSLC sprints
- Creating a question board
- Breaking down your work
- Adding to organizational knowledge
- Avoiding pitfalls