Learn how to persist data RDD actions.
- [Instructor] So, once we've calculated…some aggregations, we've filtered things,…we've built a really good dataset,…it's a nice idea to save that…so we can actually refer back to it.…So, we're going to look now at actually saving our results.…And we're going to do this using…something known as the DataFrameWriter…and this is what will allow us to save our DataFrame…as a table in Spark.…If you're connected to Hive…it will create it as a managed table…in the Hive metastore as default.…So, what we're going to do is just review…and rebuild our DataFrame,…save it as a table…and then browse the table in DataBricks.…
Here in my DataBricks environment…I have my exercise file 2.5 Saving your Results loaded.…So, the first thing we need to do…is to attach this to our cluster…and then we're just going to rerun this first cell here…which will actually read in our dataset…and create our DataFrame,…then we're going to modify that a bit…and create the top products in the UK,…and lastly, we need to actually create…a new DataFrame that we're going to use to then save.…
- Understanding Spark
- Reviewing Spark components
- Where Spark shines
- Understanding data interfaces
- Working with text files
- Loading CSV data into DataFrames
- Using Spark SQL to analyze data
- Running machine learning algorithms using MLib
- Querying streaming data
- Connecting BI tools to Spark
Skill Level Intermediate
1. Introducing Apache Spark
2. Analyzing Data in Spark
3. Using Spark SQL to Analyze Data
4. Running Machine Learning Algorithms Using MLlib
5. Real-Time Data Analysis with Spark Streaming
6. Connecting BI Tools to Spark
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.