Understand the problem statement for the use case and what the expectations are for the solution.
- [Instructor] Now that we have learned…about using Spark and Kafka for building data pipelines,…let us solve a real world problem.…This is a use case project where we first…define the problem, then architect the solution,…followed by implementing code that does the work.…For the sake of keeping this code from dragging too long,…we will purposefully focus on only a few metrics.…Now this solution can easily be scaled…for larger data elements and complex computations.…
An executive vice president of a multinational company…wants to have a real-time dashboard…in his office that would show him…the current state of affairs of his company.…The data in the dashboard will be available in real time…and will refresh every five seconds.…So this is how the dashboard is expected to look like.…It has numbers and graphs that will keep ticking.…Given that this course is focused…on spot data engineering,…we're not going to show…how this dashboard is going to be coded,…but we will show how to assemble data…that is required for this dashboard in real time.…
- What is data engineering?
- Spark and Kafka for data engineering
- Moving data with Kafka and Kafka Connect
- Kafka integration with Apache Spark
- How Spark works
- Optimizing for lazy evaluation
- Complex accumulators
Skill Level Advanced
Big Data Foundations: Program Managementwith Alan Simon1h 11m Intermediate
1. Data Engineering Overview
2. Moving Data with Kafka
3. Spark High-Performance Processing
4. Use Case Project
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.