Join Dan Sullivan for an in-depth discussion in this video SQL for DataFrames, part of Introduction to Spark SQL and DataFrames.
- [Instructor] There are a couple of different ways … of working with DataFrames. … One way is to use the DataFrame API. … And basically that is structured around using methods … on DataFrame objects. … For example, let's assume we have a DataFrame … called df for short. … There is a method on the DataFrame object called select. … And I can select a column by putting the column name … in double quotes, and passing that to the select method. … So in this example, I have a DataFrame called df, … I'm calling the select method, … and I'm asking for the CPU utilization column. … Now I also want to display the results, … so I use the .show method. … Now, like in SQL, you can do things … like group by a particular column. … So in this example, I have a DataFrame called df, … I'm applying the group by method, … and I'm telling PySpark that I want to group by the server ID. … And after I do the grouping, I'd like to do a count, … so I'd like to count the number of rows in each group. … And then I'd like to show the results. …
- Installing Spark and PySpark
- Setting up a Jupyter notebook
- Loading data into DataFrames
- Filtering, aggregating, and saving data
- Querying and modifying DataFrames with SQL
- Exploratory data analysis
- Basic machine learning
Skill Level Intermediate
1. Introduction to Spark DataFrames
2. Installing Spark
3. Getting Started with Spark DataFrames
4. SQL for DataFrames
5. Data Analysis with Spark
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.