Leveraging RDDs can be useful in certain circumstances and might be critical in maintaining older Spark code. In this video, learn how to perform basic Spark tasks using RDDs.
- [Instructor] This time we don't need…the reported crimes dataset,…but the police station's dataset.…I've provided the commands to get this dataset below.…Let's do our normal checks.…I've got the Spark software so I'm good now let me set up…the environment for this notebook.…And I'm going to store the police station's dataset…in the csv file police-station.csv.…
Now I'm actually going to be using the SparkContext sc…to open that csv file.…So let's say se.…textFile.…And it's called police-stations.csv.…And this is going to be an rdd so let's call it ps…for police station rdd…equals se.textFile.…
And let's view the first row of our rdd,…psrdd….first.…And we can see that the first row is all of the column names…so this is the header for our rdd.…So let's call it that, so ps police station header…equals psrdd.first.…Now if you want to grab the rest of the rdd…we can do so by saying psrdd…filter…and we use a lamba function so lambda line…where the line is not equal to the header.…
So ps…header.…Let's call this rdd…ps…rest.…And let's view the first row of this rdd.…
- Benefits of the Apache Spark ecosystem
- Working with the DataFrame API
- Working with columns and rows
- Leveraging built-in Spark functions
- Creating your own functions in Spark
- Working with Resilient Distributed Datasets (RDDs)
Skill Level Intermediate
1. Introduction to Apache Spark
2. Technical Setup
3. Working with the DataFrame API
5. Resilient Distributed Datasets (RDDs)
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.