Learn how big data systems are actually harder to work with and can make even basic analytical questions more difficult to answer.
- [Ben] Let's talk a little bit here about the myth that big data will actually make your analytics easier. In an ideal world, all of your data flows into your data warehouse, and that data warehouse is your big data platform. This can take a long time, however, and until all of the data sources are mirrored or moved over to your big data platform, analysts are going to have to look in additional locations to find data to get their job done. Now, data analysts are the ones on the front lines, and are generally a resourceful bunch that will find the data they need to answer their users' questions, regardless of where it lives.
But adding another system to the mix will, at least in the short term, delay them slightly, because now they're not sure where to get the data from. In my own experience, working in modern analytics platforms, it is common for my clients to need at least a half-dozen data sources just to answer a single question. Most of the modern analytics tools can handle this, so it's becoming more of a normal occurrence to not have all the data in one single location. That's not to mention that not all big data platforms support the languages or tools that our analysts are used to using.
While you're doing your needs assessment, it will be important for you to make sure and consider how your users, in this case the data analysts, will want to access the data. Because if you don't have that box checked, then putting all your data into a big data platform really isn't going to do any favors for your analytics group. And remember, they're the ones really driving home the value of your entire data science program.