Architect a solution for the use case using Kafka and Spark.
- [Instructor] Now that we know the parameters…of the Use Case, we can begin to architect a solution.…We will create an Executive Summary Table…in MySQL that will provide summaries data…that can be used to display the dashboard.…This table has one row, for every date / hour combination.…So that's 24 records a day showing totals for each hour.…This keeps the table small and manageable.…It has one column for each type of metric…that needs to be displayed,…sales, web hits, tweets, and positive tweets.…
The job of the data pipeline is to update this table…in real time.…Priming for this table that is creating the base record…for each hour, is a separate task that can be done…by editing.…So let's start with the architecture now.…First on the right side, we have a MySQL database…containing the Executive Summary Table…that is used to display the dashboard.…We start off with the pipeline for garment sales.…
The garment sales system writes sales data in real time…to its own MySQL database.…A Kafka Connects clip listens to this sales data.…
- What is data engineering?
- Spark and Kafka for data engineering
- Moving data with Kafka and Kafka Connect
- Kafka integration with Apache Spark
- How Spark works
- Optimizing for lazy evaluation
- Complex accumulators
Skill Level Advanced
Big Data Foundations: Program Managementwith Alan Simon1h 11m Intermediate
1. Data Engineering Overview
2. Moving Data with Kafka
3. Spark High-Performance Processing
4. Use Case Project
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.