Learn about the unique challenges faced in building real-time big data pipelines.
- [Instructor] In this video…we will review the challenges we face…with real-time big data processing.…The first challenge we face with big data…is the four Vs of big data itself,…velocity, variety and veracity.…Each of these four Vs poses challenges…in a big data pipeline stability…to handle them at real-time speeds.…The second challenge…is the differing throughput capabilities…of different components in a pipeline.…
Let me elaborate.…Typically, you have a producer of data,…a consumer of data…and a stream that flows from the producer…to the consumer.…In case of batch processing,…the producer can write to a storage…like a file system or database…and the consumer can read from that storage.…This means the producer and consumer…are decoupled and they can work at their own speeds…without hampering each other's capability…but in real time, data needs to flow directly…from the producer to the consumer.…
If the consumer cannot accept data…at the same speed as the producer is producing,…it creates back pressure…and brings down the entire pipeline.…
There is no coding involved. Instead you will see how big data tools can help solve some of the most complex challenges for businesses that generate, store, and analyze large amounts of data. The use cases are drawn from a variety of industries, including ecommerce and IT. Instructor Kumaran Ponnambalam shows how to analyze a problem, draw an architectural outline, choose the right technologies, and finalize the solution. After each use case, he reviews related best practices for real-time streaming, predictive analytics, parallel processing, and pipeline management. Each lesson is rich in practical techniques and insights from a developer who has experienced the benefits and shortcomings of these technologies firsthand.
- Components of a big data application
- Big data app development strategies
- Use cases: fraud detection and product recommendations
- Technology options
- Designing solutions
- Best practices