Learn how to finesse the earlier architecture blueprint for the website product recommendations use case with technology options and finer details to arrive at the final architecture.
- [Instructor] Let us now refine the architecture…we laid out before with our technology options…we discussed in the earlier video.…We will use Apache Kafka as the clickstream queue.…Kafka should be set up with multiple partitions.…Ordering of events is important within a user session,…but not across user sessions.…Make sure that the ecommerce system writes events…from one browser session to the same Kafka partition…to ensure correct sequencing of events.…
We will use Apache Spark…as the recommendation machine learning engine.…It will use machine learning to predict products…for the user in question based on the clickstream model.…Each event is handled and predicted inside the map function.…You will also access the in-memory database…from within the map function to ensure parallelism.…We will use Kafka again for the recommendation queue.…We will build a custom recommendation service…that will provide a REST-based web service…to the ecommerce system.…
This service should be stateless…and store any state-related information…
There is no coding involved. Instead you will see how big data tools can help solve some of the most complex challenges for businesses that generate, store, and analyze large amounts of data. The use cases are drawn from a variety of industries, including ecommerce and IT. Instructor Kumaran Ponnambalam shows how to analyze a problem, draw an architectural outline, choose the right technologies, and finalize the solution. After each use case, he reviews related best practices for real-time streaming, predictive analytics, parallel processing, and pipeline management. Each lesson is rich in practical techniques and insights from a developer who has experienced the benefits and shortcomings of these technologies firsthand.
- Components of a big data application
- Big data app development strategies
- Use cases: fraud detection and product recommendations
- Technology options
- Designing solutions
- Best practices