Learn how to query streaming data in real time using SparkSQL.
- [Instructor] All right, at this point,…we've already setup our stream, we have our data loaded,…we need to now actually just query our streaming data.…So, first, we're going to recreate the basic analysis,…then we'll create our in-memory SQL table,…and lastly, we'll query that table.…Here in Databricks, I have the exercise file 5.3 loaded,…and if you've already executed a few of these first steps,…you don't need to redo that,…and you can check here by running this third cell,…but if you weren't following along…with the previous clips that's fine,…execute the first one,…which will download this sales_log.zip file,…and then unzip that file so you can actually see the data…and all the different files it has there.…
Make sure also that you're connected to a cluster…and that cluster is running Spark 2.0 or greater.…Okay, so once we have that, let's just make sure…that the directory exists in my environment here…and I'll hit play,…and it looks like it's there.…So, this is where the data gets downloaded to,…databricks/driver/sales_log,…
- Understanding Spark
- Reviewing Spark components
- Where Spark shines
- Understanding data interfaces
- Working with text files
- Loading CSV data into DataFrames
- Using Spark SQL to analyze data
- Running machine learning algorithms using MLib
- Querying streaming data
- Connecting BI tools to Spark
Skill Level Intermediate
1. Introducing Apache Spark
2. Analyzing Data in Spark
3. Using Spark SQL to Analyze Data
4. Running Machine Learning Algorithms Using MLlib
5. Real-Time Data Analysis with Spark Streaming
6. Connecting BI Tools to Spark
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.