Use event-time-based data to generate analytics in Flink. Review the operation flow and how watermarks work through an example.
- [Educator] In this video, I will continue … from where we left off in the previous video. … We will use event time to do some processing. … We are going to use the audit trail … with event timestamps to create one-second summaries … and print them to the console. … One of the key problems to address, … while using event timestamps is: … what to do with late data, … what happens if an event arrives late beyond its watermark … and its window has expired. … Flink allows to capture these events as a side output … and handle them separately. … To do that, we start off by designing … a side output tag for the late audit trail. … Next, me move to the processing pipeline. … First, we convert the audit trail into a tuple … with the event timestamp and count. … We do a time window of one seconds to aggregate. … Note that these time windows are … based on event time and not processing time. … We then send the late data to the … side output tag, as defined earlier. … The late events will automatically get collected …
- Streaming with Apache Flink
- Using the DataStream API for basic stream processing
- Working with process functions
- Windowing and joins
- Setting up event-time processing
- State management in Flink
Skill Level Advanced
1. Apache Flink
2. DataStream API
4. Event Time Processing
5. State Management
6. Use Case Project
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.