Traditional datasets are organized into rows and columns. In this video, learn how to work with columns of data, including determining the range of possible values.
- [Instructor] In the next few videos,…I'll provide both the pandas Syntax and the PySpark Syntax.…If you're familiar with pandas,…it'll make the transition from pandas to PySpark easier.…In PySpark, it's possible to access a data frame's column…either by attribute, so a dot notation, or by indexing,…where we would use square brackets.…We can't always use the dot notation…because this will break when the column names…have reserved names or attributes to the data frame class.…If you've used pandas, similar rules apply…and you can use the df column…or the df with square brackets.…
In both PySpark and pandas, df dot column…will give you the list of the column names.…In both PySpark and pandas, you can select…more than one column using a list within square brackets.…In PySpark, it's more common to use data frame dot select…and then list the column names that you want to use.…Let's say you wanted to add a new column to your data frame,…where the values in this new column…are twice that of an existing column.…In PySpark, you can use the with column function.…
- Benefits of the Apache Spark ecosystem
- Working with the DataFrame API
- Working with columns and rows
- Leveraging built-in Spark functions
- Creating your own functions in Spark
- Working with Resilient Distributed Datasets (RDDs)
Skill Level Intermediate
1. Introduction to Apache Spark
2. Technical Setup
3. Working with the DataFrame API
5. Resilient Distributed Datasets (RDDs)
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.