Out of the box, Spark has many powerful functions for working with data. In this video, learn more about the optimized features of the built-in Spark functions
- [Instructor] With PySpark, the built in functions…are available from the PySpark.sql.functions library.…All you need to do is to import them…so that you can use them.…With pandas, many of these functions are available to you…and you just need to enter the data frame,…the series or column name, and then the function.…So if you want to do some string operations…in PySpark, there are options…for substring, upper, and lower.…If you're looking at date functions…then you can get the day of the week,…the day of the month, and so on.…And if you're looking at mathematical functions…then you have the option for the sine,…the cosine, log and so on.…
We can also create our own functions…called user defined functions…and we'll get into this later.…Let's head over to our notebook…to look at a few examples.…So we've listed the first five rows of our data frame.…Let's head over to the built-in function section.…So remember that we need to import the functions…that we want, so from PySpark.sql import functions…and I'm just going to show you…
- Benefits of the Apache Spark ecosystem
- Working with the DataFrame API
- Working with columns and rows
- Leveraging built-in Spark functions
- Creating your own functions in Spark
- Working with Resilient Distributed Datasets (RDDs)
Skill Level Intermediate
1. Introduction to Apache Spark
2. Technical Setup
3. Working with the DataFrame API
5. Resilient Distributed Datasets (RDDs)
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.