Learn about the use cases in this course and the process followed to solve those use cases.
- [Instructor] Let me describe to you our general approach for use cases in the rest of the course. The use cases go from simple to complex use cases. We will slowly build and add complexity throughout the course. We will focus on using existing technologies to build architectures. Our goal is reuse and avoid building what already exists. Big data is not just machine learning.
A number of students think machine learning and predictive analytics when they talk about big data. Big data architectures focus on managing and processing large amounts of data. While enabling predictive analytics is a key goal, it is not the only goal. Please remember that the focus of this course is not machine learning but software architecture. We will architect systems that enable machine learning but let data scientists worry about algorithms and such.
We will work on the use cases in this course by following a systematic process. We first analyze the problem we try to solve and set up goals for the architecture. We outline a solution for the architecture by identifying various modules, their purpose, and their interactions with other modules. We review existing technologies available for each module, evaluate them against our goals, and select the best possible options.
We then finish our architecture outline with the technology options we have chosen and finalize the blueprint. We look at designing some key elements in the architecture, we focus on scalability, performance, and usability of the solution, and how various components, technologies, need to be used for this use case. In each use case we review generic best practices for one of the components in a big data pipeline.
Here is a note about technologies used in this use case. We consider and analyze a number of big data technologies during the course. The student is expected to have some prior knowledge of them. This is actually an impossible ask since there are too many technologies involved, but we do recommend that you read up on these technologies to understand some basics. As an architect, you are expected to have some knowledge of the various technologies before you can start using them in your architecture.
There is no coding involved. Instead you will see how big data tools can help solve some of the most complex challenges for businesses that generate, store, and analyze large amounts of data. The use cases are drawn from a variety of industries, including ecommerce and IT. Instructor Kumaran Ponnambalam shows how to analyze a problem, draw an architectural outline, choose the right technologies, and finalize the solution. After each use case, he reviews related best practices for data acquisition, transport, processing, storage, and service. Each lesson is rich in practical techniques and insights from a developer who has experienced the benefits and shortcomings of these technologies firsthand.
- Components of a big data application
- Big data app development strategies
- Use cases: archiving audit logs and performing customer analytics
- Technology options
- Designing solutions
- Best practices