Learn how to use the spark-csv package to import data into a DataFrame.
- [Narrator] Let's take a look now…at working with CSV data in DataFrames.…Remember, a DataFrame is similar to the table in SQL,…Pandas in Python, or a data frame in R.…It allows for more expressive operations on data sets.…What we are going to do here is find some CSV data…then we are going to sample that data,…and then create a DataFrame with the CSV.…I'm going to switch over to the…DataBricks community edition now,…and here I have loaded from the exercise files 2.3,…and it's in a Python notebook.…Here in the Databricks environment,…I have 2.3 from the exercise files loaded.…
I need to attach this to my cluster,…and then I'm going to go ahead and execute my first cell…which shows us where some data lives.…This is under the online retail data.…This data comes along with Databricks,…so it should be a similar result that you see here.…Down below we just need to specify the path…as we've done before.…When we read that data in,…we use something different than before.…Here, we are using Spark instead of SC…for the spark context.…
- Understanding Spark
- Reviewing Spark components
- Where Spark shines
- Understanding data interfaces
- Working with text files
- Loading CSV data into DataFrames
- Using Spark SQL to analyze data
- Running machine learning algorithms using MLib
- Querying streaming data
- Connecting BI tools to Spark
Skill Level Intermediate
1. Introducing Apache Spark
2. Analyzing Data in Spark
3. Using Spark SQL to Analyze Data
4. Running Machine Learning Algorithms Using MLlib
5. Real-Time Data Analysis with Spark Streaming
6. Connecting BI Tools to Spark
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.