Users are able to create their own functions in Spark. In this video, learn about the advantages and limitations of creating functions beyond the ones built into Spark by default.
- [Instructor] You won't be surprised to hear…that you can also create your own functions,…and these functions will be applied…across your data row by row.…Now, there's a really good reason…why we focused on the built-in functions first…before looking at these UDFs, or user-defined functions.…This is because there are performance considerations…that we need to be aware of.…Now, once you create your function,…you need to register it with Spark…for it to be used on all the worker nodes.…It doesn't matter what language you've used,…so that's if your code is in Python, Scala,…or any other supported language,…the next thing Spark needs to do is…to serialize the function on the driver…and then send it out to all of the executor processes.…
When you use the function…and it's written in Scala or Java,…you can use it within the Java virtual machine,…or JVM.…This means there's little performance penalty.…You still don't get the full benefits…that Spark has for the built-in functions…that we looked at earlier,…but there isn't a bit hit on the performance.…
- Benefits of the Apache Spark ecosystem
- Working with the DataFrame API
- Working with columns and rows
- Leveraging built-in Spark functions
- Creating your own functions in Spark
- Working with Resilient Distributed Datasets (RDDs)
Skill Level Intermediate
1. Introduction to Apache Spark
2. Technical Setup
3. Working with the DataFrame API
5. Resilient Distributed Datasets (RDDs)
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.