Join Dan Sullivan for an in-depth discussion in this video Timeseries analysis with DataFrames, part of Introduction to Spark SQL and DataFrames.
- [Instructor] We're going to work … with our utilization data again, but instead of doing … just kind of a general exploratory data analysis … like we could do with any data set. … We're going to take a look at things we can specifically … do with timeseries data, and timeseries data is data that … has a set of measures and a timestamp associated with them. … Now in the case of the utilization data, … the measurements come at regular timed intervals. … So that makes it easier to work with in some ways. … So what I'm now doing is loading the data, … and I'm going to load the utilization data … and I've also created the utilization table … so we can work with Spark SQL right away. … Okay so we're going to start with Spark SQL … and let's create a select statement … and let's select the server ID and then we'll get … the min of CPU utilization … and the max of CPU utilization … and the standard deviation of CPU utilization, … and let's continue this on the next line … and of course this will be from the utilization table, …
- Installing Spark and PySpark
- Setting up a Jupyter notebook
- Loading data into DataFrames
- Filtering, aggregating, and saving data
- Querying and modifying DataFrames with SQL
- Exploratory data analysis
- Basic machine learning
Skill Level Intermediate
1. Introduction to Spark DataFrames
2. Installing Spark
3. Getting Started with Spark DataFrames
4. SQL for DataFrames
5. Data Analysis with Spark
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.