Join Alan Simon for an in-depth discussion in this video Looking at real-time data warehousing, part of Transitioning from Data Warehousing to Big Data.
- Since the beginning, data warehousing and Business Intelligence have been dominated by insights into what happened in the past. For the first 10 years or so of the data warehousing era almost all BI was strategic in nature. We would report and analyze past results from the sales organization, how our products were doing out in the marketplace, the productivity of our workforce, or any other area of our business. In the early 2000s though, strategists not only wanted visibility into the past, but also the present.
What became known as Operational Business Intelligence became popular in many organizations in which the objective was to report and analyze current results in time to take action and actually affect the outcome rather than look after the fact, so hopefully we could do a better job next time. With Operational BI, we would look at how our production-line was actually doing in the middle of a shift versus its quota. We would examine customer activity, our systems and our networks looking for intrusion or other areas of our business, again, more operationally than strategically, so we could hopefully impact the outcome in a positive manner.
Operational Business Intelligence needs data as quickly as possible, but the problem is that data warehouses aren't nearly timely enough to meet those needs. This issue of data latency was addressed in a number of different ways. In some cases, the batch ETL feeds were sped up as much as they possibly could be so daily feeds became hourly feeds, hourly feeds were done several times an hour, and in some cases this was enough to help bring data in fast enough to perform Operational BI.
In other cases though, this wasn't sufficient and entirely new systems were built around real-time messaging rather than our traditional batch TL feeds. In some cases, spinoff solutions, real-time data marts, were constructed that had no relationship whatsoever with enterprise data warehouses or any other data marts in the environment. And those did do a good job at addressing specific needs but they weren't necessarily part of an overall architected solution across the enterprise.
We did a good job at increasing ingestion speed of data, but it came at a price with architectural mismatch across the enterprise. As Big Data has become more popular and more powerful, it actually has done a great job at addressing these issues of real-time Operational Business Intelligence where one of our three Vs for velocity has played an important role in providing Operational BI to organizations.
- Exploring big data, Hadoop, and analytics
- Examining the shortcomings of traditional data warehousing
- Comparing big data architectures for next-generation data warehousing
- Understanding alternatives
- Building a roadmap
- Managing big data-driven projects
- Monitoring and measuring success