Join Dan Sullivan for an in-depth discussion in this video Basic machine learning with DataFrames, part 2, part of Introduction to Spark SQL and DataFrames.
- [Instructor] We're going to look at another … commonly used machine learning technique … or data science or statistics technique … called linear regression. … Linear regression is useful when you have data … in which you believe you can make predictions … about one variable using knowledge about another variable. … So for example, if you believe that you think … knowing CPU utilization will alow you … to guess what the number of sessions are … or the free memory are, … then linear regression would be a good technique … to use to implement that. … So once again, we'll use utilization data. … And I'll just load that. … And as in the previous video, we're uploading some code … from the machine learning libraries in Spark. … And in particular, we're loading the VectorAssembler, … which we have seen before, … and then we're also loading the linear regression models. … So, what we want to do is create a vector … with the feature columns that we're interested in. … Now for our first task, …
- Installing Spark and PySpark
- Setting up a Jupyter notebook
- Loading data into DataFrames
- Filtering, aggregating, and saving data
- Querying and modifying DataFrames with SQL
- Exploratory data analysis
- Basic machine learning
Skill Level Intermediate
1. Introduction to Spark DataFrames
2. Installing Spark
3. Getting Started with Spark DataFrames
4. SQL for DataFrames
5. Data Analysis with Spark
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.