Join Dan Sullivan for an in-depth discussion in this video Basic DataFrame operations, part of Introduction to Spark SQL and DataFrames.
- [Instructor] Now let's look at … some other basic Dataframe operations. … Now from here on out, … I will start by having the data already loaded. … Now we could do all of these exercises in a single notebook. … But I am separating them out … so that you can jump into any lesson … without having to depend on steps from previous lessons. … So each of the exercise files … will have all of the commands that are needed … to load the data that is required … for that particular exercise. … In this case, … I have our JSON file … with the utilization data already loaded. … Let's just take a quick look at that. … I've loaded the data into a Dataframe called df2. … And I want to show the first 10 rows. … So again, this has our event date time, … free memory, cpu utilization and so on. … And let's just check the count. … And there's 500,000, which we expected. … Now sometimes you might want to work with or view … the list of column names. … To do that, … you can specify the name of the Dataframe, …
- Installing Spark and PySpark
- Setting up a Jupyter notebook
- Loading data into DataFrames
- Filtering, aggregating, and saving data
- Querying and modifying DataFrames with SQL
- Exploratory data analysis
- Basic machine learning
Skill Level Intermediate
1. Introduction to Spark DataFrames
2. Installing Spark
3. Getting Started with Spark DataFrames
4. SQL for DataFrames
5. Data Analysis with Spark
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.