Learn how to build custom Spark accumulators to store and move complex data structures.
- [Instructor] Accumulators in Spark are one…of the most powerful features that Spark provides.…But the standard accumulators that come with Spark…only allow single values, like an integer value…or a string value, to be passed around…between the driver program and the notes.…So what if you want to pass multiple values?…What if you want to pass around the structure…or a class or a list?…You can do so by writing…your own accumulator implementation.…
And I'm going to show you how you can do that.…So I created my own class called CustomAccuMap…as a part of this package,…and the way you do with this, this is us…by extending the AccumulatorV2 interface from Spark.…Now, what I want to do is I want to have…an accumulator, which is a map of string and double,…so it is a map, but the key is a string,…and the value is a double,…so I can implement anything like that I want.…
So the way you would write this accumulator…is you extend AccumulatorV2,…and you'll see the input and the output…are of the same data type.…Map,<String,Double>, Map,<String,Double>.…
- What is data engineering?
- Spark and Kafka for data engineering
- Moving data with Kafka and Kafka Connect
- Kafka integration with Apache Spark
- How Spark works
- Optimizing for lazy evaluation
- Complex accumulators
Skill Level Advanced
Big Data Foundations: Program Managementwith Alan Simon1h 11m Intermediate
1. Data Engineering Overview
2. Moving Data with Kafka
3. Spark High-Performance Processing
4. Use Case Project
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.