Learn how to set up a multibroker cluster on a local machine, simulating a more realistic environment.
- [Instructor] Now we're going to get a little but more advanced and set up a multi broker cluster. And this is a more real world scenario, whereas before we were just really looking at the basic ways of sending messages through Kafka and how to set that up. So I already have Zookeeper and Kafka running, that's still going from earlier, and what I want to do now is create some config files to use for my different brokers. So I'm going to open up a new terminal window. Make that a little bit bigger so you can read it.
So here what I want to do is create new property files for each of my brokers. And I'm going to copy those from the template that is already in my config directory. So make sure that you're in that directory, so we need to go over, and if you were following along, I have it in Downloads, and then Kafka. And again, the version I'm using may be different than the one you have. So make sure you're just using whatever that folder is where you extract that. Now, under here I have a config folder. And I can just copy this from there where I use the CP command and I copy server.properties, and by giving it a space and a separate name, I'm essentially making a copy of it with a new name.
I can do that once, and then I'll just go change it again for the second one. If I want to verify those are there, I'll take a look, I'll do LS config, and enter. And you can see I have sever one ans over two properties along with a bunch of other property files. But, the ones I'm interested in there are server-1 and server-2. Now we need to update these a little bit. And we need to give them some new names, some new directories, so that way they don't conflict with each other. So we'll take a look here, and what I want to do is do VI config server-1.properties.
And I need to change some things here. So down where it says broker ID=0, I'm going to change that to broker ID=1. I'm going to look for listeners, and I'm going to change this to be listeners plain text 90 93. And I need to update the log directory. The log directory for here will be Kafka-logs-1.
That way they're going to a separate place. And remember, in a production environment, you may do this on separate disks entirely. Once I've updated these three settings, I'll go ahead and hit escape, and then W Q. Which saves those. Now I'm going to do the same for my server two properties file. Here, instead of broker 1, I'm going to make it broker 2. So I'll hit I so I can insert and adjust the text here. Change the broker ID to two, change the listener here to 90 94, for the port.
And make sure that the log directory is Kafka-log-2. Hit escape. W Q, to save the changes. Now we need to start the two nodes in separate terminal windows. We already have Zookeeper and Kafka running, but what we want to do here is set up these two new servers. So I'm just going to start one here and then do one in another window. So, I'll do bin Kafka server start, and then pass in the properties there.
And you can see that going. I'll create a new tab for a new terminal window here, and I'll do the same thing passing in the second properties file. All right, so I have my new brokers up and running. I'm going to go ahead and minimize this for now. I'll do a new terminal window, and resize it a little bit. And now I'm going to create a new topic. And this topic, unlike the one we did before, is going to be replicated to all three nodes.
So we're going to jump over to the directory where we have Kafka. Downloads, Kafka, and I'll run the Kafka topics script. I'll tell it to create a topic with a replication factor of three, with one partition, and I'm going to call it my replicated topic. So copy this from the exercise file or you can just type it in. Okay, the topic was created. Good news. Now we can see the stats about it.
Using the describe command. So we're calling that same script, Kafka topics, we're using describe, telling it where Zookeeper is, and telling it what topic to look at. So, here you can see the partition count, the replication factor, and the configs, if I make my window a little bit bigger it's easier to read. You can see the topic, the partition, the leader, and the replica is two, zero, and one. Now if we take a look at this compared to our test topic we can run the same command except with test, and you can see again the partitions zero, leaders zero, replica zero, so this is how we can set up a multi broker cluster all here on our machine with a topic that actually is replicated, so something you would typically see in a real production environment.
- Understanding the Kafka log
- Creating topics
- Partitioning topics across brokers
- Installing and testing Kafka locally
- Sending and receiving messages
- Setting up a multibroker cluster
- Testing fault tolerance