Join Alan Simon for an in-depth discussion in this video Monitoring and measuring success, part of Transitioning from Data Warehousing to Big Data.
- Transitioning from tradition data warehousing to big data is a monumental effort, and without a doubt everyone in your organization needs to know where you stand along the way, and one of the best ways to keep everyone on board and also to counteract any lingering discontent is to clearly indicate how well you're doing in your effort and the progress you've made. On the other hand failing to do so could be a significant problem for you even if your accomplishments with big data and the analytics are on or ahead of schedule. You have two techniques that you should use: Key Performance Indicators, or KPIs, and Maturity models.
Let's look at each of these. You want to use KPIs to set big data program objectives for both the business and IT sides of your organization. For example, you want to measure and track how actively and widely your big data environment and analytics are being used versus whatever you have in place right now, for example, fragmented data marts and other analytical solutions. Beyond how actively your environment's being used you also want to measure and track how many key business decisions are actually being made from your new technology, whether those decisions are made by first-line managers or if they're making their way all the way up to the executive ranks in the Board room.
And doing so will help you clearly communicate the linkage between the investments you've made and the positive impact of those investments. Keeping our concept of prescriptive analytics in mind you also wanna measure and track not only the decisions being made but the actions being taken from those decisions, again for the purposes of aligning your investments with the results. Your KPIs need to be quantifiable, not subjective. Here are some specific examples from both the business and IT sides where you can track numbers as well as look at trending over time.
Make sure that when you develop your specific big data KPIs you keep the following items in mind. Make sure you have realistic benchmarks against which you're measuring every one of your Key Performance Indicators, you don't just produce numbers, you need to understand and be able to report if you're on target, ahead of target or behind target. You need to measure both static performance at a particular point in time as well as trends, so you can understand not only how you're doing against your targets but if you're improving against those targets.
Your KPIs should follow the exact same continuum as the business-facing analytics that you built. You want to have some that look into past performance, others that are real time measures, and still others that are predictive in nature. Then you should also apply your data mining discovery analytics against program data that goes into those KPIs to help identify any interesting and important insights into how well your big data program is doing. KPIs aren't meant to be kept in a log desk, they should be widely reported and share so that everyone can get on board with the value of the big data program.
And then finally, KPIs aren't only to be reported and then forgotten, you should actively manage to what your KPIs are telling you, and then adjust aspects of your big data and analytics program accordingly. Your other major tool for measuring your program success will be Maturity models. We need to formally designate a finite number of levels along a particular Maturity curve. You need to understand where your organization is at right now and then place yourself at the appropriate level.
You need to monitor the progress as you move from level to level. Maturity models allow you to not only have the quantifiable aspects of Key Performance Indicators but also subjective criteria that are difficult or even impossible to measure. And then finally, the roadmap for big data and analytics within your organization should be well-aligned with progress along those Maturity models. For example, you may have five levels that make up a Maturity model within your organization, and as your organization moves along the Maturity curve you check off the criteria at each level and then move forward to the next level.
Here's an example of a simple five-level Maturity model in which you can see the different criteria from level zero through level four. At level zero, where you might begin, you don't have any centralized enterprise data warehouse, what you do have though are inconsistent data marts and late reports, so your organization is not very analytically/data mature. Your next stop is level one, in which you begin to implement an enterprise data warehouse but at the same time you see conflicts between the data warehouse and the existing data marts.
Still though you start to implement some predictive analytics and you proceed to the point where 10% of your data marts that you started out with are now decommissioned and retired and no longer conflicting with your enterprise data warehouse. At level two your EDW and Hadoop co-exist, they're just not integrated together, still you're making progress with predictive analytics with 25% of them reaching the CxOs, the executives within your organization, for key decisions that are made.
You move next to level three, in which you now have a Hadoop staging area that is gradually feeding your enterprise data warehouse. At this stage, for example, 75% of the data marts you started out with are now decommissioned and retired, and many of your analytics are now following a formal workflow all the way from the data-driven insights through the decisions and then actions that are taken. At level four, the most mature level, you now have a single Hadoop-based EDW with greater than 50% of the analytics all across that environment now being predictive or discovery in nature, not just after the fact reporting.
When you build Maturity models for your organization here are some critical success factors to keep in mind: Make sure that you tailor them to your organization. You need to have a formalized path to move from one level to the next. You need to make sure you have realistic targets as you move from one level to the next. You also want to have specific target timelines to move from one level to the next one. And finally, just as with Key Performance Indicators, progress along the Maturity curve for your organization needs to be actively managed and widely reported.
Both of these tools, Key Performance Indicators and Maturity models, are essential for any effort to move from traditional Data Warehouse into a new era of big data, and should be actively used to the greatest extent possible.
- Exploring big data, Hadoop, and analytics
- Examining the shortcomings of traditional data warehousing
- Comparing big data architectures for next-generation data warehousing
- Understanding alternatives
- Building a roadmap
- Managing big data-driven projects
- Monitoring and measuring success