Learn how to use the Spark shell.
- [Voiceover] One of the most important aspects of Spark…is its use of Resilient Distributed Dataset,…or RDD,…to accomplish fault tolerance.…Once created,…RDD can be transformed into another RDD,…or you can also take an action on an RDD.…Let's create our first RDD…of the README file stored…in our Spark directory.…You can see the README file in…the /usr local Spark directory.…
So let's quit Spark for now,…and let's checkout the README file, type ls.…And the README.md file is there.…Let's start the Spark shell again.…Let's call our first RDD text file…type val textFile = spark.read.textFile("README.md") .…
Press Enter.…Looks like it worked.…Now let's take some actions to…the newly-created RDD,…type textFile.first() .…Press Enter.…This action returns the first item in the dataset,…which is # Apache Spark.…
So the line appears first in the README.md file.…Let's take another action.…Type textFile.count() .…Press Enter.…This action counts the number of items in the dataset,…which is our README.md file.…There are 103 items or lines in the README.md file.…
Author
Jungwoo Ryoo
Released
8/30/2018- Enabling technologies in data science
- Cloud computing and virtualization
- Installing and working with Proxmox, Hadoop, Spark, and Weka
- Managing virtual machines on Proxmox
- Distributed processing with Spark
- Fundamental applications of machine learning
- Distributed systems and distributed processing
- How Hadoop, Spark, and Weka can work together
Skill Level Beginner
Duration
Views
-
Introduction
-
Course organization1m 17s
-
1. Introduction to Data Science
-
Introduction1m 51s
-
Data science2m 53s
-
Fundamental skills3m 42s
-
Enabling technologies2m 4s
-
-
2. Cloud Computing
-
Cloud fundamentals3m 29s
-
Types of cloud3m 19s
-
Solution providers2m 22s
-
Proxmox: Installation2m 26s
-
3. Distributed File Systems
-
Distributed file systems2m 44s
-
Fundamentals2m 45s
-
Hadoop hands-on2m 8s
-
Hadoop: Preparation4m 11s
-
Hadoop: Installation4m 18s
-
Hadoop: MapReduce hands-on8m 52s
-
-
4. Distributed Processing
-
Spark: Installation6m 24s
-
Spark: Spark shell4m 28s
-
Spark: pyspark4m 32s
-
Spark: Application4m 1s
-
5. Machine Learning
-
Machine learning2m 41s
-
Fundamentals2m 16s
-
Types of machine learning2m 59s
-
Weka: Installation2m 33s
-
Weka: GUI3m 35s
-
Weka: Training vs. testing3m 21s
-
Weka: Clustering2m 12s
-
-
6. Case Study
-
Putting it all together2m 42s
-
Hadoop cluster: Operation4m 57s
-
Spark, YARN, and Hadoop6m 42s
-
Weka and Spark3m 12s
-
-
Conclusion
-
Next steps41s
-
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.
Share this video
Embed this video
Video: Spark: Spark shell