Learn how to finesse the earlier architecture blueprint for the real-time fraud detection use case with technology options and finer details to arrive at the final architecture.
- [Instructor] Let us now finish the outline…for real-time fraud detection in this video.…We will use Apache Kafka as the streaming queue.…In order to achieve the expected response times…of a few minutes, Kafka should have a minimum…or no backlog at all times.…This can be achieved by creating enough partitions…on Kafka and Apache Spark to de-queue…at the speed this is queued.…The optimal partition size need to be figured out…to benchmarked testing.…
Next comes our prediction engine.…The prediction engine to be used would be Spark.…We should combine Spark streaming…with prediction to de-queue…and manage process in real time.…As each micro batch comes in,…a map function needs to be called…inside which each individual transaction will be processed…and the model upload.…It would be a good idea…to load the model in each of the partitions…in Apache Spark.…The output queue is also going to be Apache Kafka.…
Given that the percentage of fraudulent transactions…should be less than one percent,…this queue would typically be empty most of the time.…
There is no coding involved. Instead you will see how big data tools can help solve some of the most complex challenges for businesses that generate, store, and analyze large amounts of data. The use cases are drawn from a variety of industries, including ecommerce and IT. Instructor Kumaran Ponnambalam shows how to analyze a problem, draw an architectural outline, choose the right technologies, and finalize the solution. After each use case, he reviews related best practices for real-time streaming, predictive analytics, parallel processing, and pipeline management. Each lesson is rich in practical techniques and insights from a developer who has experienced the benefits and shortcomings of these technologies firsthand.
- Components of a big data application
- Big data app development strategies
- Use cases: fraud detection and product recommendations
- Technology options
- Designing solutions
- Best practices