Understand where big data can displace legacy systems and where it probably shouldn't.
- [Josh] When I talk with clients about big data platforms, and we try to understand the use cases, a lot of times they want to look for cost-saving measures. And so there's this myth that Big data will eliminate the need for X, whatever X may be, some other platform, some other thing they currently do. And I want to think about this by understanding what Big data is, and what it is not. Now Big data is unstructured, in a way, or semi-structured. The data comes in in various formats. Different file types, sometimes there's a structure like a JSON structure, or sometimes there's XML, other times it's just pure Log data, pure text.
Big data is also a huge volume of data, we're talking Petabytes or more. So when you think about Big data replacing something, unless you have something already that's operating at this scale, it's probably not going to be displacing any of the other stuff. Now I will add that data bases have evolved to be more like these Big data platforms, but because it's a Big data system know that there are many, many platforms involved, many different things going on. And in fact, one of the downsides of many Big data platforms is that it makes it more difficult for your analyst to actually find answers sometimes because of the speed and the complexity of the system itself.
Now Big data is not trans-actual in nature. So if you have a transactional system, say an order processing system, Big data systems generally aren't going to support that. There are some ways to get around that and to kind of add additional layers to provide that effect, but unless you need to, that's really not a place you want to go. The systems also are not simple or easy to deal with. They're complex, there's a lot of configurations, and a lot of things that frankly can go wrong. Along with that, we don't really typically think of Big data systems as a structured data warehouse.
So there is data there and you can create structures, but unless you really are going a full scale operation down this path, typically it's not going to replace your structured version of your data warehouse. And remember, it's not just one thing, Big data platforms are many many things, they're entire ecosystems of platforms. And so what we want to think about then is that what are all the pieces that we need and some of them while they may replace other things that we already have, often they're just actually increasing our capabilities.
So when we think about Big data, and we think about the ability for it to replace something, I would say that's generally a myth. It may replace a few things, but what we'd want to think about is how Big data systems can actually add value to our business. And do things that our other systems cannot. Of course finding cost-saving measures are great, but if you go into it with that mindset, I'm sorry to say that I believe you're going to be disappointed.