Learn how to create DataFrames by loading external data files.
- [Instructor] In this video,…I'm going to create data frames.…Now, data frames, they're kind of like relational tables.…They're a data structure that's organized into rows,…and they have named columns.…Now you may have heard of data frames before…if you've worked with R/R or the pandas package…in Python.…The data frames in Spark are very similar.…Now, what I'd like to do here,…is create three data frames using some text files…that are available as exercise files.…So I you have access to those exercise files,…you can download the text files and follow along with me.…
The first thing I'll do is create a value,…or a local variable, called spark,…which is a Spark session.…So the first thing I'm going to do for that is…import a package that we'll be using,…and that package is called…org.apache.spark.…sql.SparkSession,…and then I'll create a value called spark,…and I'll assign that to a session,…and I'll call the builder function,…and I'll give this session a name,…I'll call it DataFrameExercise,…and then we'll ask Spark to get it,…
Dan also focuses on using Scala with Spark, a distributed processing platform. He first describes how to work with Resilient Distributed Datasets (RDDs)—a fundamental Spark data structure—and then explains how to use Scala with Spark DataFrames, a new class of data structure specially designed for analytic processing. He wraps up the course by providing a summary of advantages of using Scala for data science.
- The advantages of Scala for data science
- Scala data types
- Scala arrays, vectors, and ranges
- Parallel processing in Scala
- Mapping functions over parallel collections
- When and when not to use parallel collections
- Using SQL in Scala
- Scala and Spark RDDs
- Scala and Spark DataFrames
- Creating DataFrames
Skill Level Intermediate
Java for Data Scientists Essential Trainingwith Charles Kelly2h 43m Intermediate
1. Introduction to Scala
2. Parallel Processing in Scala
3. Using SQL in Scala
4. Scala and Spark RDDs
5. Scala and Spark DataFrames
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.