Join Dan Sullivan for an in-depth discussion in this video Exploratory data analysis with DataFrames, part of Introduction to Spark SQL and DataFrames.
- [Instructor] Now that we've seen, … how to work with Spark data frames, using the data frame API … and Spark SQL we can now start to look at how we use … those tools for some higher level tasks, … like exploratory data analysis and machine learning. … So in this video we'll look at how to use the data frame API … for some basic exploratory data analysis … with the utilization data we've looked at previously. … So, at this point I have opened a new Jupyter Notebook … and I have done the preliminary specifications … and data loading, so the data has been loaded. … The first thing I'd like to do is create a table … which is accessible from SQL and to do that I'll specify … our data frame called df util … and I'll call the createOrReplaceTempView method … to create a table which I'll call utilization, … and let's just verify the count on df util. … It should be 500,000 and it is so we're all set. … Now the first thing I'd like to show you … is an API command called describe, … and you can describe a data frame by specifying …
- Installing Spark and PySpark
- Setting up a Jupyter notebook
- Loading data into DataFrames
- Filtering, aggregating, and saving data
- Querying and modifying DataFrames with SQL
- Exploratory data analysis
- Basic machine learning
Skill Level Intermediate
1. Introduction to Spark DataFrames
2. Installing Spark
3. Getting Started with Spark DataFrames
4. SQL for DataFrames
5. Data Analysis with Spark
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.