Understand the various stages of data engineering and the factors to consider while designing applications for these stages.
- [Instructor] Data engineering involves…taking data through a series of steps…before they become consumable by end users.…This is called a data pipeline.…The goal of data engineering is to architect…and build pipelines that provide functionality,…speed, scalability, and reliability required…by the business to use data effectively.…The various stages in a pipeline are acquisition,…transport, storage, processing and servicing.…
The first stage in data engineering is data acquisition.…Data acquisition modules focus on sources of data.…The engineering challenges in this stage…involve around answering the following questions.…What is the format of source data?…Data could be raw bytes,…text files, databases.…How best to acquire them?…What interfaces are available to acquire these data?…This can be standard protocols like JDBC,…HTTP, RES, et cetera.…
Or they could be custom based on the origin.…Are there security issues…in acquiring data from these sources?…This includes authentication,…authorization and encryption concerns.…
- What is data engineering?
- Spark and Kafka for data engineering
- Moving data with Kafka and Kafka Connect
- Kafka integration with Apache Spark
- How Spark works
- Optimizing for lazy evaluation
- Complex accumulators
Skill Level Advanced
Big Data Foundations: Program Managementwith Alan Simon1h 11m Intermediate
1. Data Engineering Overview
2. Moving Data with Kafka
3. Spark High-Performance Processing
4. Use Case Project
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.