From the course: Apache Flink: Real-Time Data Engineering
Join today to access over 22,600 courses taught by industry experts or purchase this course individually.
Watermarks - Flink Tutorial
From the course: Apache Flink: Real-Time Data Engineering
Watermarks
- [Instructor] Event times are the most appropriate times, to use for windowing. But, how do we know, if all the events for a given time window, has arrived at the processing function? What if some of them are still in transit, while we compute summaries for the window? We can solve the problem using, watermarks. Watermarks are timestamps, that are used in event time processing. They determine when the events are actually processed. Events that arrive at the processing node, will wait for the watermark to happen, before they can be processed. When a watermark happens, all events that arrived from the previous watermark, to this watermark, will get processed. Watermarks can be either periodic, or they can be based on custom logic. Watermarks can also have delay buffers. Buffers allow for a delay in processing, to account for latency from the source, to the processing function. Here is an example of using watermarks.…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.