Join Dan Sullivan for an in-depth discussion in this video Save data from DataFrames, part of Introduction to Spark SQL and DataFrames.
- [Instructor] Now sometimes after we've been working … with DataFrames, and creating new DataFrames, … and running calculations, and doing sampling … and so forth, we might want to save our results out. … It's very simple way to do that. … Now, here, I've started with a DataFrame already loaded, … df1 has our location temperature data. … If I want to save that out, … I can simply specify the name of the DataFrame, … that's df1, and then use the write method … and specify the csv method within that, … and then specify a name for what I'd like to save. … In this case, it's df1.csv. … And I'll execute that. … So what that's done is it has saved the DataFrame … out to disk using the csv format. … So let's take a look at the directory. … So, to execute a shell command from within … the Jupyter notebook, you can use the exclamation mark, … and that basically tells Jupyter notebook, … everything that's going to follow … is basically a shell command. … So I can do things like ls for list, … and then specify df1.csv. …
- Installing Spark and PySpark
- Setting up a Jupyter notebook
- Loading data into DataFrames
- Filtering, aggregating, and saving data
- Querying and modifying DataFrames with SQL
- Exploratory data analysis
- Basic machine learning
Skill Level Intermediate
1. Introduction to Spark DataFrames
2. Installing Spark
3. Getting Started with Spark DataFrames
4. SQL for DataFrames
5. Data Analysis with Spark
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.