From the course: Cloud Hadoop: Scaling Apache Spark

Unlock the full course today

Join today to access over 22,600 courses taught by industry experts or purchase this course individually.

Spark Streaming

Spark Streaming - Apache Spark Tutorial

From the course: Cloud Hadoop: Scaling Apache Spark

Start my 1-month free trial

Spark Streaming

“

- [Instructor] So now we're going to look at a streaming example inside of Databricks. So we're going to bring in our streaming notebook. And this is for Spark streaming for processing of data. Not ingest, processing. So here we are in the streaming notebook. As we get started, the first thing that we're doing is verifying we have Spark 2.0, because the Streaming API has been added in this version. So a quick way to do that is to check for the Spark context, of course, we have to attach. And we're fine here. Then we're going to load some sample data. Now first, we're going to just do batch processing and then we're going to see how we change our code to do streaming. And of course, like everything else, this is in Python. So we are loading some JSON data here and now we're going to do batch processing. So we're going to define a static data frame on the files that should be kind of familiar by now. This is similar to…

Contents