Join Dan Sullivan for an in-depth discussion in this video Querying DataFrames with SQL, part of Introduction to Spark SQL and DataFrames.
- [Instructor] Up to now, we've been using … the Spark DataFrame API to work with DataFrames. … Now, we're going to switch gears and we're going to work with SQL. … In particular, we're going to use Spark SQL … for working with DataFrames. … As in previous videos, I'm started with data already loaded. … Let's just quickly go through the steps … that are involved with that. … First, I import a pyspark library … that allows us to work with SQL. … I create a Spark session global variable … which allows us to work with a distributed Spark session. … Then I've defined a string that points to my directory … which holds my data. … And then, I create another string which points … to the file I want to load, … and then I execute a Spark read command … specifying the JSON format. … And then finally, I've listed out the first 10 rows … of this DataFrame, which I called df. … Let me briefly explain some of the columns. … In this DataFrame, we have utilization data … about fictional servers. … And so we measure things like CPU utilization, …
- Installing Spark and PySpark
- Setting up a Jupyter notebook
- Loading data into DataFrames
- Filtering, aggregating, and saving data
- Querying and modifying DataFrames with SQL
- Exploratory data analysis
- Basic machine learning
Skill Level Intermediate
1. Introduction to Spark DataFrames
2. Installing Spark
3. Getting Started with Spark DataFrames
4. SQL for DataFrames
5. Data Analysis with Spark
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.