Learn how to monitor the Kafka cluster.
- [Instructor] Monitoring in Kafka is pretty robust. I'm going to show you the latest and greatest here. There is an older API that uses something called Yammer metrics where it spits out a ton of details, and statistics about your cluster. But all of those metrics as well as a framework that you can use to do long-running tests and regression analysis have been packaged up and released by LinkedIn into something called Kafka Monitor. This is an open-source component that you can use to run these long-running tests, and monitor your Kafka Cluster.
Before, the problem was that you may be looking at something right now and everything's fine. But what happens is some of these problems creep up over time. So they're not immediately obvious. You need to set up tests that can be run over longer periods of time, to see the impact in the overall cluster. In the Kafka Monitor, what a user can do is set up these different tests, and have services that use those tests to interact with your cluster. They give feedback to the user in this long-running test.
They give you that ability to do regression analysis and see how the performance is being affected in other parts of the system as well as over time. A good example might be if you're trying to develop a new app, and it's going to be doing a lot of stuff with a specific topic or a number of topics. You can design the test to have it run, and see if it's going to cause any issues on your cluster. This is also a great way to do continuous deployment, where when you run your tests, you can also be deploying the new code if the tests are successful.
If you're getting data back from your Kafka Cluster and from the Kafka Monitor, that say that this isn't going to cause any problems. This is a really nice way to take a look and do some really advanced testing on your cluster to ensure the health of it over time. Another interesting component of the Kafka Monitor is its ability to test across data centers. If you have multiple clusters living in different data centers, you can have the Kafka Monitor set up a service that actually mirrors these tests across them to see if there's any issues when you actually do this cross-data center operation.
It's not that common that you have Kafka running active across multiple data centers, because latency and throughput are the names of the game. If you have things separated too far, it can really slow things down and cause some issues for your applications using Kafka. The Kafka Monitor is a great new platform that you can use in your system, and it's open-sourced from LinkedIn, and a great way for you to monitor what's going on and ensure the longevity and health of your cluster overall.
- Understanding the Kafka log
- Creating topics
- Partitioning topics across brokers
- Installing and testing Kafka locally
- Sending and receiving messages
- Setting up a multibroker cluster
- Testing fault tolerance