Discuss the features provided by Spark that make it ideal for big data engineering.
- [Instructor] We will now explore what makes Spark…ideal for data engineering.…So why is Spark the best data engineering tool available?…It is because it handles the unique challenges posed…by big data very well.…First, it is a big data compute engine that is…built ground up.…It is not an older tool that is masquerading…as a big data engine but one where big data issues have been…thought through and incorporated in its base architecture.…
It provides faster data processing than MapReduce…since it uses memory biz data structures…and minimizes network traffic.…It also provides massive horizontal scalability.…You can keep adding more and more modes to handle…more and more data loads without having…to recode your obligation.…It provides support for streaming, which enables you…to do real time big data engineering and provides real…time results results on big data.…It supports machine learning libraries, which round…out the complete suite of capabilities provided by Spark.…
And finally, there is a big list of third-party integrations…
- What is data engineering?
- Spark and Kafka for data engineering
- Moving data with Kafka and Kafka Connect
- Kafka integration with Apache Spark
- How Spark works
- Optimizing for lazy evaluation
- Complex accumulators
Skill Level Advanced
Big Data Foundations: Program Managementwith Alan Simon1h 11m Intermediate
1. Data Engineering Overview
2. Moving Data with Kafka
3. Spark High-Performance Processing
4. Use Case Project
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.