Learn about the concept and importance of Spark as a real-time distributed processing engine and its different modes of operation.
- Unlike MapReduce, Spark is capable…of stream processing.…Stream processing refers to realtime handling of data.…It's ideal when you need instant feedback…from a data analytics tool.…Let's say that you are developing an anomaly detection tool.…In this scenario, you cannot afford to wait…until the end of the week when your batch job…is supposed to run.…You need to respond to an anomaly immediately.…
What makes Spark fast is it's in-memory processing.…All the processing is done in the main memory,…which is much faster than storage devices,…but the drawback is the cost.…The memory chips are more expensive than the hard drives.…MapReduce is a default distributed processing…solution for Hadoop, but Hadoop allows you…to use Spark instead.…In fact, there are multiple ways you can…use Spark with Hadoop.…
The first option is using Spark in a standalone mode.…In this mode, you can run Spark alongside…an existing Hadoop installation.…To access HTFS from Spark, you just need an HTFS URL.…The second option is using YARN,…which was introduced from Hadoop 2.0.…
- Enabling technologies in data science
- Cloud computing and virtualization
- Installing and working with Proxmox, Hadoop, Spark, and Weka
- Managing virtual machines on Proxmox
- Distributed processing with Spark
- Fundamental applications of machine learning
- Distributed systems and distributed processing
- How Hadoop, Spark, and Weka can work together
Skill Level Beginner
Course organization1m 17s
1. Introduction to Data Science
2. Cloud Computing
3. Distributed File Systems
4. Distributed Processing
5. Machine Learning
6. Case Study
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.