Join Dan Sullivan for an in-depth discussion in this video Filtering DataFrames with SQL, part of Introduction to Spark SQL and DataFrames.
- [Instructor] Next, we're going to take a look … at how to filter data frames using Spark SQL. … So I've started a new workbook, … I've imported the SQL library from PySpark, … I've created a Spark session, … and then loaded my data from the JSON file. … Again, we're using the utilization data, … which includes measurements on CPU utilization, … free memory, and session count, … and that's organized by time and by server ID. … So the first thing I want to do, since I want to … work with SQL, is to create or replace a temp view … based on the data frame. … So I'll specify the data frame, df, … and then specify create or replace … temp view, … and specify the name of the table that I'd like to use. … In this case we'll use utilization again. … Now I can execute a Spark SQL statement, … and I'm going to save the results as another data frame, … and I'll call that df_sql, … and to create that data frame, … I will invoke Spark SQL with a SQL command, … and I'm going to use select star from utilization …
- Installing Spark and PySpark
- Setting up a Jupyter notebook
- Loading data into DataFrames
- Filtering, aggregating, and saving data
- Querying and modifying DataFrames with SQL
- Exploratory data analysis
- Basic machine learning
Skill Level Intermediate
1. Introduction to Spark DataFrames
2. Installing Spark
3. Getting Started with Spark DataFrames
4. SQL for DataFrames
5. Data Analysis with Spark
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.