Join Dan Sullivan for an in-depth discussion in this video Exploratory data analysis with Spark SQL, part of Introduction to Spark SQL and DataFrames.
- [Instructor] Now, we saw how we could get … things like the count the mean … standard deviation using the DataFrame API. … Let's do that with Spark SQL. … And to do that, we'll specify Spark SQL, … and then we'll give it a command. … In this case it'll be SELECT; … let's select min of CPU utilization … and the max of CPU utilization … and the standard deviation of CPU utilization. … And we'll specify from utilization, … because that's the table we specified … with our create or replace tempview, … and let's be sure to show this, … because the result is a data frame. … And so we have here our minimum CPU utilization … is about 22%, max is 100%, … and the standard deviation is about 15, … which is what we saw up above, … so no surprises there. … I am going to just copy this command … and we'll make it a little easier to read; … I'm going to make this multi-line. … I'll use this backslash, we'll say FROM utilization, … and now we want to specify our group by clause. … So to do that I can continue in line …
- Installing Spark and PySpark
- Setting up a Jupyter notebook
- Loading data into DataFrames
- Filtering, aggregating, and saving data
- Querying and modifying DataFrames with SQL
- Exploratory data analysis
- Basic machine learning
Skill Level Intermediate
1. Introduction to Spark DataFrames
2. Installing Spark
3. Getting Started with Spark DataFrames
4. SQL for DataFrames
5. Data Analysis with Spark
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.