Learn about the number of strategies to handle the challenges in building real-time big data pipelines.
- [Instructor] So what are the…various strategies we can use for…real-time big data processing?…Here are a few.…We start off by discussing the…differences between synchronous…and asynchronous pipelines.…A synchronous pipeline is one…where the source places are request…and waits for the response.…The pipeline needs to process the request…and get back with the response…while the source is waiting.…For example, a user clicks a button…on a website, and waits for the recommendations.…
An asynchronous pipeline is the one…where the source places the request,…but does not wait for a response.…The source moves on to do other work.…The pipeline, after processing the request,…will push the results back to the source asynchronously.…For example, when a user clicks…a button on a website, he is moved to the next page.…The pipeline would process recommendations asynchronously.…An AJAX script on a browser will pull the pipeline.…
When the recommendations are ready,…they a provided to the AJAX script…which will then ping the browser…
There is no coding involved. Instead you will see how big data tools can help solve some of the most complex challenges for businesses that generate, store, and analyze large amounts of data. The use cases are drawn from a variety of industries, including ecommerce and IT. Instructor Kumaran Ponnambalam shows how to analyze a problem, draw an architectural outline, choose the right technologies, and finalize the solution. After each use case, he reviews related best practices for real-time streaming, predictive analytics, parallel processing, and pipeline management. Each lesson is rich in practical techniques and insights from a developer who has experienced the benefits and shortcomings of these technologies firsthand.
- Components of a big data application
- Big data app development strategies
- Use cases: fraud detection and product recommendations
- Technology options
- Designing solutions
- Best practices