Join Dan Sullivan for an in-depth discussion in this video Load data into DataFrames: CSV Files, part of Introduction to Spark SQL and DataFrames.
- [Instructor] First thing I'm going to do is load a CSV file. … And I have a file called location_temp, … which is a time series file which contains … locations of sensors and the temperatures taken … at particular periods of time. … So I'm going to create another variable called file_path, … which is equal to my data_path plus the name of my file, … which is location_temp.csv. … And I'm just going to hit Return, … so notice it does not execute that command yet. … Now, I'm going to create a data frame, which I'll call df1. … And I'm going to set df1 to the results of reading that file … and I'm going to use a Spark read command called spark.read. … And I'm going to specify the format … and I'm going to specify CSV. … Now, there are a number of different ways of expressing … how to read from a CSV file. … I'm using this particular format right now. … And I'm going to pass in an option, … which says the header is true. … And I want to load from … my file path. … So that's going to read a file, …
- Installing Spark and PySpark
- Setting up a Jupyter notebook
- Loading data into DataFrames
- Filtering, aggregating, and saving data
- Querying and modifying DataFrames with SQL
- Exploratory data analysis
- Basic machine learning
Skill Level Intermediate
1. Introduction to Spark DataFrames
2. Installing Spark
3. Getting Started with Spark DataFrames
4. SQL for DataFrames
5. Data Analysis with Spark
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.