Build code for processing tweet data in the solution.
- [Instructor] In this video,…I'm going to show you how to set up the pipeline for tweets.…As we discussed before,…this company has a social media analytics application,…which is going to keep listening tweets for this company…and writing out these tweets to log files.…So we're going to set up a file stream source…to those log files that receive data,…and then process it and come up with the comfort tweets…and also the positive tweets.…Let us now focus on setting up Kafka Connect.…
The Kafka Connect setup is for a file source,…and we would be looking at a file called Tweet Start Text,…which has the actual tweets.…So the content of the file looks something like this.…So it has a timestamp in the beginning,…and then it has the actual tweet text.…So we keep it pretty simple for this specific exercise,…and that has already been copied onto the right bottom,…if you can see a file called Tweet Start Text…under source ware, so that is already set up.…Now let us go and set up the tweet-file-source.properties…for our Kafka Connect.…
- What is data engineering?
- Spark and Kafka for data engineering
- Moving data with Kafka and Kafka Connect
- Kafka integration with Apache Spark
- How Spark works
- Optimizing for lazy evaluation
- Complex accumulators
Skill Level Advanced
Big Data Foundations: Program Managementwith Alan Simon1h 11m Intermediate
1. Data Engineering Overview
2. Moving Data with Kafka
3. Spark High-Performance Processing
4. Use Case Project
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.