Join Doug Rose for an in-depth discussion in this video Rely on serendipity, part of Learning Data Science: Using Agile Methodology.
- You've seen that it can be difficult to have clearly defined objectives with your data science team. Still many organizations find it difficult to even imagine working without clear objectives. You see objectives everywhere. Almost every management book talks about how to set objectives. You set career objectives. Even when you're training, there are clearly defined learning objectives. These objectives guide much of what we do, but they might not be as valuable as you think. There's been some interesting work in this area over the last few years.
You'd think that questioning objectives would come from some new age social science, but it's actually coming from where you'd least expect. It's coming from the world of machine learning and artificial intelligence. These are the same people who are working to have computers display intelligent behavior. They're finding that so much of what we know is based on unplanned discovery. We actually learn more from our wandering than we do from our set objectives. One of the best books on this topic is Why Greatness Cannot be Planned: The Myth of the Objective, by Ken Stanley and Joel Lehman.
Professor Stanley runs a research group at the University of Central Florida that works on artificial intelligence. In the book, he says that, objectives actually become obstacles towards more exciting achievements, like those involving discovery, creativity, invention, or innovation. Remember that this is coming from a leading computer scientist. It's not an infographic quote from Deepak Chopra. The way you should think about this is that the more you focus on clearly defined objectives, the less likely you are to make interesting discoveries.
Everyone on the data science team should be comfortable with creative wandering. In fact, Professor Stanley points out that the team should rely on pure serendipity. Serendipity is a strange word to put in a book on artificial intelligence. Serendipity is when something just happens. It's unpredictable or unplanned. It's like when you bump into an old friend and then decide to sit down and have a cup of coffee. It's unscheduled, unplanned, and unpredictable. As strange as it might sound, data science has to rely on some serendipity.
Sometimes a team member will see something that they weren't expecting. It might look interesting or unusual. It's important for the team to follow up on that discovery. You don't want to focus on an objective, at the expense of this area of exploration. Professor Stanley calls these stepping stones. These are the interesting things that eventually lead to insights. If you ignore them then you're likely to miss key discoveries. Imagine that we're back in our running shoe website. The data science team's objective is to predict how many new sales the site can expect in the upcoming year.
While looking at the data, the analysts see something interesting. There's a slight dip in sales on Sundays over the last few weeks. If the team were focused on the objective, they might have ignored this interesting discovery. It's hard to imagine that the slight dip, would help them predict upcoming sales. A data science team should follow up on this interesting little discovery. It might not lead to anything. In fact, most of these little discoveries, will just be dead ends. Still a few of them, will be stepping stones, to something that could be very valuable.
The more the team explores the data, the more they'll create these connections. Still all this language is something you don't typically hear in organizations. Words like, stepping stones, serendipity, and discovery. It sounds more like a trailer for a romantic comedy. Yet, these are key parts of learning something new and interesting. In fact, a report from the patent office suggests that almost half of all discoveries are the result of simple serendipity. A team was looking to solve a problem, and then someone's insight led them in an entirely new direction.
Much of discovery is being comfortable with not knowing where your information will lead. You have to be able to pursue the unexpected. These stepping stones are only clear at the end of your path. The key is to not ignore the things that look interesting, just to stay true to your objectives.
This course shows how to structure your work within a two-week sprint. See how to work within a data science life cycle (DSLC)—a methodology for cycling through questions, research, and reporting every two weeks. Explore key practices to help your team break down the work so it fits within a two-week sprint. Learn how to use tools like question boards to encourage discussion and find essential questions. And most importantly, learn how to grow your team's shared knowledge and avoid common pitfalls.
- Defining data science success
- Determining project challenges and criteria for success
- Using a DSLC
- Iterating through DSLC sprints
- Creating a question board
- Breaking down your work
- Adding to organizational knowledge
- Avoiding pitfalls