Do an exercise to use Apache Spark Streaming to subscribe to and digest Kafka data.
- [Instructor] In this video, I am going to show you…how to use a part of spark to subscribe to a topic…and then do some processing on the topic.…To begin with, I'm going to show you this ETC,…this Windows System 32 drivers etc hosts file.…I'm adding an entry on line 23 for the specific IP address…and forwarding it to quickstart.cloudera.…If you don't have this,…sometimes the program will just get stuck…and fail silently,…so just make sure you add this to your ETC host file.…
Now, moving on to the example.…The example is available…in your spark big data engineering project.…The file is called spark kafka streaming JDBC example.java.…So let's go and explore how this code looks like.…All the example files in this specific project…are done with the main…so that they can all run independently.…However, when you write code,…of course you're going to be launching separate threads,…or classes and stuff like that for this one.…
So to begin with, this is a main file.…We start off by setting up spark configuration,…so it's a new spark configuration.…
- What is data engineering?
- Spark and Kafka for data engineering
- Moving data with Kafka and Kafka Connect
- Kafka integration with Apache Spark
- How Spark works
- Optimizing for lazy evaluation
- Complex accumulators
Skill Level Advanced
Big Data Foundations: Program Managementwith Alan Simon1h 11m Intermediate
1. Data Engineering Overview
2. Moving Data with Kafka
3. Spark High-Performance Processing
4. Use Case Project
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.