Discover how to make Apache Spark work with other big data technologies to build data pipelines for data engineering and DevOps.
- [Kumaran] Hi, my name is Kumaran Ponnambalam. Welcome to my course about constructing big data engineering pipelines using Apache Spark. The course focuses on data engineering and architecture. It shows how to wire together individual components to create big data pipelines. I will delve into the theory behind data engineering and also show you use cases. These use cases demonstrate code that can be used to build data engineering pipelines.
This course is generally intended for big data engineers, architects, and developers. The code examples are in Java since it is the most popular programming language for building production quality Spark applications. So let's get started.
- What is data engineering?
- Spark and Kafka for data engineering
- Moving data with Kafka and Kafka Connect
- Kafka integration with Apache Spark
- How Spark works
- Optimizing for lazy evaluation
- Complex accumulators
Skill Level Advanced
Big Data Foundations: Program Managementwith Alan Simon1h 11m Intermediate
1. Data Engineering Overview
2. Moving Data with Kafka
3. Spark High-Performance Processing
4. Use Case Project
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.