Join Dan Sullivan for an in-depth discussion in this video Aggregate data with DataFrame API, part of Introduction to Spark SQL and DataFrames.
- [Instructor] Now let's take a look … at aggregating using the DataFrame API. … Now, I've opened a new Jupyter notebook … and as I mentioned in an earlier video, … I'm going to start with the data loaded. … Now, if you have access to the exercise files, … you'll have these commands … in each individual chapter's exercise. … So in the first step, … I defined a string which has a data path. … In the second step, I built on that data path … and created a file path and pointed to a data file … which has some location and temperature information … and then I read that into a data frame. … And then here in step three, … I'm simply showing the first 10 rows. … So we have a data frame called df1 … and it has location and temperature information … where the temperature is measured in Celsius. … Now what I'd like to do is I would like to count … how many different measurements we have for each location. … So to do that I'm going to reference the data frame … and I'm going to use the groupBy operation …
- Installing Spark and PySpark
- Setting up a Jupyter notebook
- Loading data into DataFrames
- Filtering, aggregating, and saving data
- Querying and modifying DataFrames with SQL
- Exploratory data analysis
- Basic machine learning
Skill Level Intermediate
1. Introduction to Spark DataFrames
2. Installing Spark
3. Getting Started with Spark DataFrames
4. SQL for DataFrames
5. Data Analysis with Spark
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.