Join Dan Sullivan for an in-depth discussion in this video Set up a Jupyter notebook, part of Introduction to Spark SQL and DataFrames.
- [Instructor] All right, I have opened a terminal, … and I've navigated to my working directory, … which is just in my Spark SQL directory, … I've created one called work. … And I'll just list the files, … showing it's empty right now. … So now I'm going to run PySpark, … this will start a Jupyter notebook for me. … Now because the directory's empty, … I don't have any notebooks here, … so I'm going to create a new notebook, … and I'm going to use Python3, … and the first thing I want to do is load some data. … Before I can start working the data, … I need to do a little setup work. … First thing I need to do is import … the PySpark SQL package from pyspark.sql, … I want to import the one thing that I needed … for this example, is called SparkSession, … so let's load that. … Now I want to actually create a Spark context, … and that's basically a pointer to a data structure … that represents the cluster and allows me … to send commands to the cluster … and receive information back. … So to do that, I'm going to create a variable called spark, …
- Installing Spark and PySpark
- Setting up a Jupyter notebook
- Loading data into DataFrames
- Filtering, aggregating, and saving data
- Querying and modifying DataFrames with SQL
- Exploratory data analysis
- Basic machine learning
Skill Level Intermediate
1. Introduction to Spark DataFrames
2. Installing Spark
3. Getting Started with Spark DataFrames
4. SQL for DataFrames
5. Data Analysis with Spark
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.