Brief overview on why Spark and Kafka are ideal for big data engineering.
- [Instructor] When you want to architect…a big data pipeline for your business,…it's really stupid to build everything from scratch.…Rather, you want to utilize…best-of-breed, off-the-shelf products…and focus on your application functionality.…While using off-the-shelf products for big data,…you want to choose those that provide…horizontal scalability, a wide range of functionality,…less coding, cheaper to acquire and maintain,…and a growth and support system for the product.…
With this in mind, I'm going to use the following components…in this course to build a model big data pipeline.…Apache Spark will be the data processing engine,…Apache Kafka will be the data acquisition…and transport layer,…MySQL and HDFS will provide storage capabilities.…They provide all the capabilities that we talked about…and also interface well with one another.…
While this is a recommendation,…I strongly suggest that you evaluate your situation…and use case while choosing the right products…for your architecture.…Going forward in this course,…
- What is data engineering?
- Spark and Kafka for data engineering
- Moving data with Kafka and Kafka Connect
- Kafka integration with Apache Spark
- How Spark works
- Optimizing for lazy evaluation
- Complex accumulators
Skill Level Advanced
Big Data Foundations: Program Managementwith Alan Simon1h 11m Intermediate
1. Data Engineering Overview
2. Moving Data with Kafka
3. Spark High-Performance Processing
4. Use Case Project
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.