Latency issues arise when using event time for Flink processing. Learn how watermarks help solve the latency issue and ensure accuracy of results.
- [Instructor] Event times are the most appropriate times, … to use for windowing. … But, how do we know, … if all the events for a given time window, … has arrived at the processing function? … What if some of them are still in transit, … while we compute summaries for the window? … We can solve the problem using, watermarks. … Watermarks are timestamps, … that are used in event time processing. … They determine when the events are actually processed. … Events that arrive at the processing node, … will wait for the watermark to happen, … before they can be processed. … When a watermark happens, … all events that arrived from the previous watermark, … to this watermark, will get processed. … Watermarks can be either periodic, … or they can be based on custom logic. … Watermarks can also have delay buffers. … Buffers allow for a delay in processing, … to account for latency from the source, … to the processing function. … Here is an example of using watermarks. … We are going to have some custom watermarks logic. …
- Streaming with Apache Flink
- Using the DataStream API for basic stream processing
- Working with process functions
- Windowing and joins
- Setting up event-time processing
- State management in Flink
Skill Level Advanced
1. Apache Flink
2. DataStream API
4. Event Time Processing
5. State Management
6. Use Case Project
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.