Discuss why data engineering for big data is a different ball game and what special factors to consider for such applications.
- [Instructor] So what is the difference…between regular data and the so called Big Data?…Well Big Data is classified on four attributes.…Volume, Velocity, Variety, and Variability.…A Big Data processing pipeline needs to account…for the following requirements:…Functionality, Speed, Reliability,…Security, and Availability.…Each of these four attributes pose unique challenges…while engineering Big Data pipelines…with these requirements.…
Let us now look at these challenges in detail.…Volume is the very first challenge.…Every byte of data needs resources.…CPU cycles to process, memory and disc to store,…and bandwidth to move.…The higher the volume, the more the resources you will need.…The challenge starts increasing exponentially…when a single processor or pipeline cannot handle…the expected volume.…This demands building pipelines…that can provide horizontal scalability…in order to handle volume…while maintaining latency requirements.…
The second attribute is Velocity.…In the Big Data world we are talking…about data from browsers, sensors, social media,…
- What is data engineering?
- Spark and Kafka for data engineering
- Moving data with Kafka and Kafka Connect
- Kafka integration with Apache Spark
- How Spark works
- Optimizing for lazy evaluation
- Complex accumulators
Skill Level Advanced
Big Data Foundations: Program Managementwith Alan Simon1h 11m Intermediate
1. Data Engineering Overview
2. Moving Data with Kafka
3. Spark High-Performance Processing
4. Use Case Project
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.