Learn about the term real time and what it means with respect to expected latencies.
- [Instructor] Let us start our real-time course with first setting a definition for the term real-time. So, what is real-time? A more appropriate question would be how real-time is real-time? To define real-time, we need to first define the term response time. It is the time taken for data to move from the point of data collection to the point of insight-based action.
Response time is also called latency in the same context. Here are some examples for response time in the big data world. Here is the time from when the user enters credentials in the log in page to the time when his log in is declined because the big data pipeline in the background suspects that the log in is a hack. It is the time taken from when a tweet is entered by a user on his mobile phone to the time when the sentiment of the tweet is presented to the analyst who is looking at this real-time tweets.
It is the time taken from when the user clicks a link on an e-commerce website like Amazon to the time when a set of product recommendations are provided to the user. So, how would you define response times for real-time analytics? Is it sub second, seconds or a few minutes? It depends upon the use case. It is the time delay that is acceptable without hampering user experience or analytics effectiveness.
It can range anywhere from sub second to a few hours. As an architect, you are always motivated to design for the fastest possible response times for your use case but remember that there is also a cost of response. Cost of response is the money you need to invest in the big data architecture to deliver response time. The cost typically includes hardware, software, network resources and the engineering and support organizations needed to build and manage this infrastructure.
Typically, as the response time expectations get higher, or the target response times get lower, the cost increases exponentially. That is the key reason why you need to first determine an average or maximum response time that is acceptable for the solution. You have to go as high as possible for the response time value without impacting user experience or application effectiveness.
There is no coding involved. Instead you will see how big data tools can help solve some of the most complex challenges for businesses that generate, store, and analyze large amounts of data. The use cases are drawn from a variety of industries, including ecommerce and IT. Instructor Kumaran Ponnambalam shows how to analyze a problem, draw an architectural outline, choose the right technologies, and finalize the solution. After each use case, he reviews related best practices for real-time streaming, predictive analytics, parallel processing, and pipeline management. Each lesson is rich in practical techniques and insights from a developer who has experienced the benefits and shortcomings of these technologies firsthand.
- Components of a big data application
- Big data app development strategies
- Use cases: fraud detection and product recommendations
- Technology options
- Designing solutions
- Best practices