Explore the avenues available to scale the solution for the use case.
- In this video, we are going to look at our options…for scaling our Use Case Solution.…Let us first look at the components…that are there inside our solution…that can be scaled independently.…First, you have Kafka Connect that is there.…Then, you have Kafka broker.…Then there is Apache Spark Streaming.…And finally, there is MySQL.…So let us look at each of these components one by one.…Let's start with Kafka Connect.…
How do you scale Kafka Connect?…You configure the tasks.max parameter…in your Connect configuration to the number…of tables or unique files that you are listening to.…Second, if you want to do more scaling,…you can create different instances of Kafka Connect…for each of the tables or files you are monitoring.…And also, you can increase the frequency of polling…so you're going to be looking at smaller sets…of data for each poll.…You can use Kafka Connect in the cluster mode…where you can have a number of worker note…you are doing the job for you in pablam.…
Then we come to Kafka Broker.…How can we scale Kafka Broker?…
- What is data engineering?
- Spark and Kafka for data engineering
- Moving data with Kafka and Kafka Connect
- Kafka integration with Apache Spark
- How Spark works
- Optimizing for lazy evaluation
- Complex accumulators
Skill Level Advanced
Big Data Foundations: Program Managementwith Alan Simon1h 11m Intermediate
1. Data Engineering Overview
2. Moving Data with Kafka
3. Spark High-Performance Processing
4. Use Case Project
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.