From the course: Stream Processing Design Patterns with Spark

Unlock the full course today

Join today to access over 22,500 courses taught by industry experts or purchase this course individually.

Alerts and thresholds: Pipeline implementation

Alerts and thresholds: Pipeline implementation - Apache Spark Tutorial

From the course: Stream Processing Design Patterns with Spark

Start my 1-month free trial

Alerts and thresholds: Pipeline implementation

- [Instructor] Having created an acception message generator, let's now build a pipeline for the alerts and thresholds use case. The code for this is in the Streaming Alerts And Thresholds class, under chapter three. The initial set up here is similar to the streaming analytics use case. We start off the Kafka alerts data generator in a separate thread, then we create a spark session. We use the read stream operator to read the streaming alerts input topic into the raw alerts D of data frame. Now, we need to unpack the value attribute in the Kafka message and extract the relevant information. We first remove the double quotes on the strings in the message. Then we split the CSV string based on the separator and various attributes in the string. They are now created as a data set row. We then proceed to write each message to the console using the forage writer. The start operator triggers actual execution of the…

Contents