Do an exercise to use Kafka Connect to write to an HDFS sink.
- [Instructor] In this video, I'm going to show you…how to build a HDFS sink with Kafka Connect.…The code for all of this is available in the file…code_02_03 Building a HDFS Sink.…So, to begin with we got a configured HDFS…with the connect-hdfs-sink.properties.…So, this is the configuration file for Kafka Connect,…so this has a name in line 31…and then there is a connector.class in line 34.…
As you would have noticed,…this is not a standard connector,…this is built by another party called Confluent.…The way you can get this connector…is you can download the source code,…build it yourself…and then place the jar files in the class part…or to make things easier for you I've already built it…and I made all those jar files available for you…under the directory includes inside your exercise files.…So, what you need to do is you need to create…a directory called includes…under the spark-data-engineering directory…and then copy all the files under this include folder…into this destination folder,…so that's what you need to do…
- What is data engineering?
- Spark and Kafka for data engineering
- Moving data with Kafka and Kafka Connect
- Kafka integration with Apache Spark
- How Spark works
- Optimizing for lazy evaluation
- Complex accumulators
Skill Level Advanced
Big Data Foundations: Program Managementwith Alan Simon1h 11m Intermediate
1. Data Engineering Overview
2. Moving Data with Kafka
3. Spark High-Performance Processing
4. Use Case Project
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.