Join Dan Sullivan for an in-depth discussion in this video Load data into DataFrames: JSON Files, part of Introduction to Spark SQL and DataFrames.
- [Instructor] Now, I'm back in my Jupyter Notebook homepage … and I saved out that last workbook … that we were working with. … I called it simply 03.01 Loading csv files into dataframes. … And now I'm going to create a new Notebook, also with Python 3. … And, in this example, I'd like to show you … how to read a json file. … Now, the formats going to be pretty similar. … For example, the first thing we want to do … is import from pyspark.sql and we want to import SparkSession … and then we want to create a spark context … which is the variable again that gives us a reference point … for communicating and manipulating the cluster. … And, to do that we call SparkSession … and we call the builder and within the builder … we want to call the getOrCreate command. … Now, we also want it to find our data path … and that's the same thing I used before. … And again, you'll change this … to wherever you happen to store the data files. … And, also I just want to point out when I'm hitting Return, …
- Installing Spark and PySpark
- Setting up a Jupyter notebook
- Loading data into DataFrames
- Filtering, aggregating, and saving data
- Querying and modifying DataFrames with SQL
- Exploratory data analysis
- Basic machine learning
Skill Level Intermediate
1. Introduction to Spark DataFrames
2. Installing Spark
3. Getting Started with Spark DataFrames
4. SQL for DataFrames
5. Data Analysis with Spark
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.