Join Dan Sullivan for an in-depth discussion in this video Filter data with DataFrame API, part of Introduction to Spark SQL and DataFrames.
- [Instructor] Now let's take a look … at how we can use the Dataframe API … to filter some of the rows in our Dataframe. … I uploaded the location temp csv file … and again that's time series data … that has a location ID and a temperature measurement … at a particular point and time. … Now we have a number of different locations. … So, for example, … if we wanted to see all of a particular location, … we could specify a Dataframe name … in this case df1, … and we want to apply the filter command or filter method. … And what we want to do … is we want to specify … that the Dataframe 1 location ID column … should be equal to, … let's call it loc0, location 0. … And that'll return some results … and let's make sure we show those results. … And what you'll notice here … is we got 20 rows of location 0 … but how many are there? … Let's take a look at how we can get a count … for the results of a filter. … So let's filter Dataframe 1 using the filter method. … And then we want to specify the column we're working with. …
- Installing Spark and PySpark
- Setting up a Jupyter notebook
- Loading data into DataFrames
- Filtering, aggregating, and saving data
- Querying and modifying DataFrames with SQL
- Exploratory data analysis
- Basic machine learning
Skill Level Intermediate
1. Introduction to Spark DataFrames
2. Installing Spark
3. Getting Started with Spark DataFrames
4. SQL for DataFrames
5. Data Analysis with Spark
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.