The history of Spark is important to understanding how it was built and how it is currently maintained. In this video, learn about the origins of Spark and Databricks.
- [Instructor] Now Spark started in 2009…as a research project in UC Berkeley's RAD Lab.…This later became the AMP Lab.…Now the researchers in the lab had previously…been working on Hadoop MapReduce,…so they knew that MapReduce was inefficient…for iterative and interactive computing jobs.…So right from the beginning,…Spark was designed to be fast for interactive queries…and iterative algorithms.…They incorporated ideas like support…for in-memory storage and efficient fault recovery.…In Zaharia and his team's 2009 paper,…they say that while Spark is still currently…a working prototype,…the performance results they were getting…were very encouraging.…
Even at that time, Spark could outperform…machine learning workloads by a factor of 10.…You can see this on page five of their paper.…So they did a couple of experiments.…One of them was a logistic regression job…across 20 nodes with each node having four cores.…Not only did they crash a node…to demonstrate that Spark could continue to function…with fewer nodes,…they then compared this with the Hadoop implementation.…
- Benefits of the Apache Spark ecosystem
- Working with the DataFrame API
- Working with columns and rows
- Leveraging built-in Spark functions
- Creating your own functions in Spark
- Working with Resilient Distributed Datasets (RDDs)
Skill Level Intermediate
1. Introduction to Apache Spark
2. Technical Setup
3. Working with the DataFrame API
5. Resilient Distributed Datasets (RDDs)
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.