Join Dan Sullivan for an in-depth discussion in this video Working with NA values in DataFrames, part of Introduction to Spark SQL and DataFrames.
- [Instructor] It's not uncommon to have data missing … from datasets, now when we work with Sequel, … we're used to working with nulls … and working around nulls. … When we work with data frames, … the absence of data is indicated by an NA … So in this lesson, we're going to look … at how we can work with NAs and nulls … using data frames and Spark Sequel. … So as I've done before, I've started … with some data already loaded, … so let's review what I have done already. … In this Jupyter Notebook, I have a number … of import statements, so I'm importing the Row function … which we used in the previous video, … to create a local data frame, and then I also … have Spark Session which allows us to work with Spark Sequel … Now I'm also importing a couple … of things we haven't seen before, … I'm importing a function called lit … which allows us to create a literal column for a data frame, … and I'm also importing a data type called String Type … which we'll use a little bit later. … Next, I create a data frame using data …
- Installing Spark and PySpark
- Setting up a Jupyter notebook
- Loading data into DataFrames
- Filtering, aggregating, and saving data
- Querying and modifying DataFrames with SQL
- Exploratory data analysis
- Basic machine learning
Skill Level Intermediate
1. Introduction to Spark DataFrames
2. Installing Spark
3. Getting Started with Spark DataFrames
4. SQL for DataFrames
5. Data Analysis with Spark
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.