Spark has a number of components that allow it to perform operations on data. In this video, get the high-level overview of how those pieces all work together.
- [Narrator] So let's take a look at…the different components of Apache Spark.…The Driver sits on a node on the Cluster…and does a couple of things.…It maintains information about the Spark application.…So it'll do things like respond to a users program or input…and it distributes and schedules work across the Executors.…As you can imagine, the Driver process is critical…and maintains all relevant information…for the Spark application.…The Executors on the Worker Node carries out the work…that's been assigned by the Driver and it reports back…on the state of the computation, back to the Driver.…
It's important to remember that the Driver…and the Executor are just processes.…This means that they can all exist on one machine…if you're running on local nodes so they just be threads…and they can run on different machines…if you're running on a cluster.…Finally, you have the Cluster Manager.…You can think of the Cluster Manager…as task managers or resource managers.…So, when you submit Spark applications to Cluster Managers,…
- Benefits of the Apache Spark ecosystem
- Working with the DataFrame API
- Working with columns and rows
- Leveraging built-in Spark functions
- Creating your own functions in Spark
- Working with Resilient Distributed Datasets (RDDs)
Skill Level Intermediate
1. Introduction to Apache Spark
2. Technical Setup
3. Working with the DataFrame API
5. Resilient Distributed Datasets (RDDs)
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.