Discuss some best practices while building data transport modules within the big data architecture.
- [Instructor] We will now look at some…of the best practices and considerations…for real-time streaming products in this video.…Streaming should guarantee delivery…of messages pushed into the streaming queue.…Different streaming products…offer different kinds of guarantees…like only once, at least one…and at most once.…You should consider these guarantees…and how they apply for your use case.…Ordering of messages…may or may not be important for your use case.…
In cases you need it,…it is important to choose a streaming product…that guarantees ordering messages pushed into the queue…can be de-queued in the same order.…Sometimes when failures in processing happens,…it is required to restart the streaming…from an earlier point.…Your solution will need to provide…an intermediate data storage guarantee.…One of the key purposes of the message queue…is to decouple the publishers and the subscribers…so that they can work at different rates…and the queue can balance their loads.…
The streaming solution should support failover…
There is no coding involved. Instead you will see how big data tools can help solve some of the most complex challenges for businesses that generate, store, and analyze large amounts of data. The use cases are drawn from a variety of industries, including ecommerce and IT. Instructor Kumaran Ponnambalam shows how to analyze a problem, draw an architectural outline, choose the right technologies, and finalize the solution. After each use case, he reviews related best practices for real-time streaming, predictive analytics, parallel processing, and pipeline management. Each lesson is rich in practical techniques and insights from a developer who has experienced the benefits and shortcomings of these technologies firsthand.
- Components of a big data application
- Big data app development strategies
- Use cases: fraud detection and product recommendations
- Technology options
- Designing solutions
- Best practices