Typical operational tasks

- [Narrator] Let's take a look now at some of the typical operational tasks that you're going to need to perform on your Kafka cluster. And the first one I want to talk about is adding and removing topics. So this is a real common thing, and there's a script that comes with Kafka that you can use when you're executing them from the terminal. Here what you'll see is the kafka-topics.sh, this is the script file. If you're on Windows, you want to look for the one with the dot bat extension for the batch files. You specify where Zookeeper is running and you specify create and topic, then you give it a topic name.

The next thing is the number of partitions. As we've talked about, the topics are all partitioned, so how many of those are going to exist? Then the replication factor, so how many times do you want this to be replicated? And finally, you add any config values that you need to add in addition to these. So when you're creating it, it's really pretty straightforward, by doing create, the topic, the topic name and then the additional parameters that you need to create there. You're also going to want to know how to add partitions, because as the topic grows, you may want to change the amount of partitions or even just as the number of consumers grow, you may want additional partitions to feed all the different consumers that are reading from it.

So for this one, you use that same script, the Kafka topics, and here what we want to do is use an alter command. So instead of a create topic, we're doing dash, dash alter and then dash, dash topic, giving it the topic name, and the new amount of partitions. Similarly, you can modify the topic configurations using the same script, and again you're going to say alter, the topic, give it the topic name, and then whatever the config is. You can also delete configurations similarly, but this time using the command line option of delete dash config and then the config name.

Now if you wanted to delete a topic, this is disabled by default, so you'll actually have to set this to enabled on your Kafka cluster before you can do that. And now that's because often the topics will have the data and they will live on for a long time and deleting them is just something, maybe if you're testing it out that makes sense, but a lot of times you're not going to want to delete them, you'll just kind of let them become stale and just stay there. So if you did want to do that after you've enabled it, you'll do very similar to the other one where we use the Kafka topics script, and here you do dash, dash delete, the topic and then the topic name.

You'll often want to check the position of a consumer, and that again will let you know exactly where that consumer is at in terms of what data it's read from the topic. So there's another script for this and you can see that here and we'll take a look at this in a little bit, but it's pretty straightforward. You point it at Zookeeper, and then you tell it which consumer group to actually look at and it will spit back all of the consumers in that group and the offset or the position of that topic. Lastly, you may need to configure the consumer groups, and this is when you're setting them up.

It's pretty straightforward. First if you wanted to just take a look at which consumers groups exist, you can use the Kafka consumer groups script and you can just ask it for a list of them. You point it at where the bootstrap server is and then you tell it dash, dash list. This will give you a list of them. From there, now that you know what the consumer groups are, you can then describe them and you can also configure them using that same script with the describe option and then the dash, dash group option, getting it the group name.