Do an exercise to use Kafka Connect to read from a file source.
- [Instructor] In this video,…I'm going to show you how to set up…a Kafka Connect file source…where Kafta Connect listens to a file source,…observes the data as it changes,…and publishes it as a Kakfa topic.…So, the code for this is available…in the file Code 02_02 Building a File Source.…Inside this file, let us now try to focus…on what the configuration file for Kafka Connect is.…That is starting from line number 16 and 17.…
In the configuration file,…you have to give a name to the connector on line 20.…Any name you can give it here,…in this case, it is demo-file-source.…Then you have to indicate to the connector…which class you need to use for this specific source.…In line 23, we have configured that as fileStreamSource.…Then on line 26, you have tasks.max=1.…That is the number of parallel tasks…that you want to launch,…and this provides scalability for you.…
Typically, the number of tasks you want to launch…is equal to the number of partitions…that you have configured for this Kafka topic.…Then you have on line 29 the file…
- What is data engineering?
- Spark and Kafka for data engineering
- Moving data with Kafka and Kafka Connect
- Kafka integration with Apache Spark
- How Spark works
- Optimizing for lazy evaluation
- Complex accumulators
Skill Level Advanced
Big Data Foundations: Program Managementwith Alan Simon1h 11m Intermediate
1. Data Engineering Overview
2. Moving Data with Kafka
3. Spark High-Performance Processing
4. Use Case Project
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.