Join Dan Sullivan for an in-depth discussion in this video Using Jupyter notebooks with PySpark, part of Introduction to Spark SQL and DataFrames.
- [Instructor] Now let's install Jupyter Notebook … and to do that, we open a terminal and then we … enter the command $ pyton3 -m pip install jupyter. … Now I already have it installed, but if you don't, … then this would download and install … the Jupyter files for you. … Okay, let's work with PySpark. … So I've opened a terminal window … and I've navigated to my working directory, … which in this case, is in my home directory … under LinkedIn Learning and I simply call it Spark SQL. … I can start PySpark by typing the PySpark command … and this will start Jupyter Notebook for me … and you'll notice when Jupyter Notebooks open, … it lists the contents of the directory, so … there are some data files and some IPYNB files. … IPYNB is a suffix used for Jupyter Notebooks … and that comes from an earlier version … of Jupyter Notebooks which were called iPython Notebooks. … That's why it's called IPYNB. … What I would like to do is just show you … what a basic Jupyter Notebook looks like. …
- Installing Spark and PySpark
- Setting up a Jupyter notebook
- Loading data into DataFrames
- Filtering, aggregating, and saving data
- Querying and modifying DataFrames with SQL
- Exploratory data analysis
- Basic machine learning
Skill Level Intermediate
1. Introduction to Spark DataFrames
2. Installing Spark
3. Getting Started with Spark DataFrames
4. SQL for DataFrames
5. Data Analysis with Spark
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.