Joining data from two datasets allows for richer analysis. In this video, learn how to look up values from one dataset in another dataset as you work with real data from the City of Chicago.
- [Instructor] A join brings together…two sets of data, the left and the right.…Spark compares the value of one or more keys…of the left and right data and evaluates…a join expression to decide whether…it should bring the left set of data…and the right set of data.…The join expression determines whether…two rows should join, and the join type…determines what should be the result.…Here are a couple of the different join types.…We have inner joins, outer joins, left outer joins,…and right outer joins amongst others.…
The PySpark syntax for joins is very simple.…Df is the left data frame, which is joined…with df2, the right data frame.…Df.column equals df2.column determines…whether the two rows should join…and how is the join type, so that it's an inner join,…outer join, left outer join, and so on.…Let's take a look at the different…join types in more detail.…Inner joins keep rows with keys that exist…in the left and right dataset.…
Outer joins keep rows with keys…in either the left or right dataset.…Left outer joins keep rows with keys…
- Benefits of the Apache Spark ecosystem
- Working with the DataFrame API
- Working with columns and rows
- Leveraging built-in Spark functions
- Creating your own functions in Spark
- Working with Resilient Distributed Datasets (RDDs)
Skill Level Intermediate
1. Introduction to Apache Spark
2. Technical Setup
3. Working with the DataFrame API
5. Resilient Distributed Datasets (RDDs)
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.