Learn how to create a local file and import it into a Kafka topic using the standard connector.
- [Instructor] Now, we've just been using the terminal window itself to send data through Kafka, but often what you're going to do is need to set up connectors that look for changes to things, whether it be a database, or a flat file, and it will automatically pull those in. So that's what we're going to do here, we're going to import and export some data into Kafka using a connector. I'll start by opening up a new terminal window, I'm going to resize it a little bit, so you can see it there. Now I just want to create a file here, so I'll just say echo -e, foo, backslash n for new line, and bar.
So it's going to be two lines, foo and bar. And I'll send this over to test.txt file. Now we need to set up a connector, and point it at that. So I'm going to jump over to where I have Kafka running, in the Downloads folder. And in fact, I'm going to copy that file back from my main directory, into this one here. Just to check and see it's there, you can see I have test.txt, there we go.
So now that we have our file setup, we need to set up a connector here. And all of the configs here ship with Kafka and they just kind of act as templates. So what I want to do is copy this command, which is going to build a standalone connector here. And it's going to do it to a file source, and we need then pass in also the file sink properties which are the ones that actually say where the file's going to go to. We'll see a bunch of stuff happening on the screen here. And once that's done there, the connector is now running, and essentially it's going to read in everything from test.txt.
And so now what I need to do, is create, or I can go look at the test.sink file, in which case is where the data's actually going. So let me create a new tab. And I'll just ls here, and notice there's a new file in there. test.sink.txt. So that's because what's happening is I have a connector for test.txt, which was set up by default with the connections there, and it's going to be writing the output into that file. We can take a look, and just see what's happening in that file, just by doing a cat.
And you can see both of the lines there. If I want to do the same thing on test.txt, you can see that it is there as well. So this was just a simple way to set up the config to read from that text file, and then write to another file, the sink file. We can also take a look at this data in the consumer itself. So if we look at a topic called connect-test, which is what was created with our previous statement, you'll see that we have essentially data living in our file there.
Now if I open up a separate terminal window here, and I'm going to go add a line to that file. Let me go cd, Downloads, kafka. And I'll make this a little bit bigger so you can read it. And I'm just going to copy this little guy here, which will echo Another Line into it. You can see how it popped up there, immediately in our consumer. I can keep doing this, Another Line again. And it just keeps going and going.
So what's happening is, the connector in Kafka is looking at this file, this test.txt. Any time data in it is modified, it's going to pull it into a consumer, that I created called connect-test. And it's going to write it out actually, to another file. This test.sink.txt. So these configs come with Kafka, and you can deconstruct them. But this illustrates here how Kafka can essentially be a listener for events from a data source, and then also a producer of events going into another data source, in this case just another text file.
- Understanding the Kafka log
- Creating topics
- Partitioning topics across brokers
- Installing and testing Kafka locally
- Sending and receiving messages
- Setting up a multibroker cluster
- Testing fault tolerance