Learn about the log which is central to Kafka.
- [Instructor] So what is a log when we talk about the log and the importance of it in a streaming platform? Let's take another look at how the streaming platform works. Remember, we have data providers and in Kafka terms, we call these producers and these providers make changes to categories of things or business elements such as a customer placing an order or someone registering on your website or maybe even issuing a refund for something. All of these things occur in your apps in the systems that run your business.
Now, when those systems make changes, it creates events and you can kind of think of these terms interchangeably, a change or an event. So a log is something you can think of as a never ending stream of these changes to a specific category or entity. Let's take a look at a customer information example. Now, if a customer were to create their profile on your website, that right there is a change. They've now created their profile and this is an entry into your log for all of your customer information.
Other consumers can then read that information and do something with it. So for example, if I had two systems managing customer information, the first one is the website where people can go online and create their profile and if they wanted to, they can try to place an order but at some point, maybe I want the order to go to an enterprise sales rep, maybe it's above a certain threshold. Well, the sales team probably uses something like Salesforce or another CMS to manage the customer information, the accounts and the orders.
So while the customers created this profile on the website with things like their billing information, that data also needs to exist in Salesforce if certain conditions are met and they need to work with an enterprise sales associate. So that customer information now assuming that sync occurred exists in two different places. Now, if the customer calls in and they're trying to verify the address with the sales person, maybe they found out that they typed it incorrectly and the sales person goes into Salesforce where they do their business and they update it.
Well, what's going to happen? Now, you have two different versions of that customer address living in different data sources. Well, you can try to build the sync between these two systems but how do you know which one is right? How do you know that the customer information is right and that the sales person didn't enter it wrong as well? This can create all kinds of problems and imagine if you have a company that has lots of different systems and you're trying to build these connections, these bilateral connection points between every single one of 'em. It can become an absolute mess.
Now, if you're using Kafka, this is where the log becomes really handy because the log, it becomes the true source of this information. So what we can do is we can set up a connector that any time the customer information changes inside of Salesforce, it will update the log and then the website which also has its customer information can read from that log and it'll update itself and then match Salesforce. Likewise, as new profiles are coming in and being created on the website, that can then add to the log and then Salesforce can read from the log.
So there's a way to use the log as the source of truth here and you don't have to set up individual connection points between every system that you use to run your business. Now, this is also really helpful if we wanted to say build a new application and load all of the customer information in. I don't need to know where each different part of the customer information lives. I can go to the log and it's the single place I have to find all the information about my customers.
- Understanding the Kafka log
- Creating topics
- Partitioning topics across brokers
- Installing and testing Kafka locally
- Sending and receiving messages
- Setting up a multibroker cluster
- Testing fault tolerance