Finesse the earlier architecture blueprint for the log accumulation use case with technology options and finer details, to arrive at the final architecture.
- [Instructor] Now that we have decided on the…technologies to use,…let us lay out the architecture…for the Server Log Accumulation pipeline.…We should start with the outline that we set out earlier.…We will use Flume Agents on each of individual web servers…to listen to the log directory,…acquire new log files and push them into the consolidator.…The Flume Agent will use Avro as a protocol…to push the data from the web server to the consolidator.…
The consolidator will also run a Flume Agent…to aquire data from this Avro pipes.…We will build a Flume Interceptor…which will intercept the Flume log events…on the consolidator, convert them into json format,…and publish them back to the Flume pipeline.…We will use HDFS plus Hive as the data store.…The Flume consolidator will push the json events…to this data store.…
We need to also build custom receivers…for the NMS system.…We don't know what the NMS system requires and supports,…and we leave it to its own owners,…since this is out of scope for this architecture.…
There is no coding involved. Instead you will see how big data tools can help solve some of the most complex challenges for businesses that generate, store, and analyze large amounts of data. The use cases are drawn from a variety of industries, including ecommerce and IT. Instructor Kumaran Ponnambalam shows how to analyze a problem, draw an architectural outline, choose the right technologies, and finalize the solution. After each use case, he reviews related best practices for data acquisition, transport, processing, storage, and service. Each lesson is rich in practical techniques and insights from a developer who has experienced the benefits and shortcomings of these technologies firsthand.
- Components of a big data application
- Big data app development strategies
- Use cases: archiving audit logs and performing customer analytics
- Technology options
- Designing solutions
- Best practices
Skill Level Advanced
Big Data Foundations: Program Managementwith Alan Simon1h 11m Intermediate
1. Intro to Big Data Applications
2. Use Case 1: Data Warehouse (DW)
3. Use Case 2: Log Accumulation (LA)
4. Use Case 3: IT Operations Analytics (OA)
5. Use Case 4: Customer 360 (C360)
6. Use Case 5: Customer Analytics (CA)
- Mark as unwatched
- Mark all as unwatched
Are you sure you want to mark all the videos in this course as unwatched?
This will not affect your course history, your reports, or your certificates of completion for this course.Cancel
Take notes with your new membership!
Type in the entry box, then click Enter to save your note.
1:30Press on any video thumbnail to jump immediately to the timecode shown.
Notes are saved with you account but can also be exported as plain text, MS Word, PDF, Google Doc, or Evernote.