In this video, Jeff Winesett dicusses the benefits and provides examples of using AWS SQS service to decouple application and system components. Message queues are perfect to help achieve asynchronous processes and improve system availablity and throughpu
- [Instructor] This chapter focuses on the services and tools needed to help decouple system components, reduce tight dependencies, and thereby create a system that will scale with ease. The idea behind decoupling components is to build the system such that if any single component of the system were to become unresponsive for some reason, the other components would continue to work as if no failure is happening. Amazon's Simple Message Queue is a service that helps in this regard. Amazon's Simple Message Queue Service or simply SQS, provides an implementation of message queuing software in the cloud.
It provides a reliable, highly scalable, and distributed system for passing messages between computers and application components. Message queues have been around for a long time and are a great way to get independent systems and components to communicate with one another. Message queues also provide a manner in which systems and components within systems can remain independent and autonomous. Here is an example of a tightly coupled system.
Each component calling the next component directly and synchronously. In this case, if Controller B has a problem, Controller A also has a problem because the direct calls Controller A is making to Controller B will fail. While Controller A waits on Controller B, it likely also stops accepting any new input, thereby halting the entire system. Taking advantage of the SQS service, could greatly improve this architecture.
With SQS queues in place, rather than communicating directly, the system components use queue buffers to send messages to one another. These queues allow the components to communicate asynchronously, to support concurrency, to remain highly available, and to better handle load and spikes. As a result, the overall system is stronger. Now, if Controller B fails, Controller A will still continue to perform its job and the message will buffer and wait for Controller B to start working again.
If there are a lot of messages waiting once Controller B is back online, multiple Controller B components could be added to work in parallel. These are some of the many advantages of working with message queues to decouple system components. Take for example an application that adds a watermark to an image to deter unlicensed use of images and protect digital ownership rights. Such an application has the following workflow... First, the user uploads an image to the server, then the image is downsized to a smaller image, then a watermark is added to the image, and finally a user is notified via email that the watermarked image is ready for use.
SQS could play a prominent and important role in this application's architecture. When the user uploads an image to the web server, the web server can send a message to an SQS queue indicating the image is ready to process. A consuming application that is polling the queue for new messages, receives the message, and can proceed with all of the processing. Resizing, applying the watermark, and sending a notification. This is a step in the right direction, as this does decouple the web request from the rest of the image processing, but it does not decouple enough.
If something breaks with the watermarking routine, all resizing and notifications stop as well. SQS could be used further. The resizer process should work independently of the watermarking process, and each should work independently from the notification process. Message queues can be used to achieve the needed communication between them. Here is another approach that makes further use of SQS. The user still starts off uploading an image to the server and the server sends a message notifying the system a new image was uploaded, which kicks off the process.
The components that resize the image are notified, which results in starting the resizing process. When done, another message is sent to notify the watermarking components. They add the needed watermark and send a message indicating completion of the watermark and for the system to send out an email notification to the user. Now, there are four types of components in this system... Web servers to handle the uploads, resizers to handle the image resizing, watermarkers to handle the watermarking, and notifiers to handle the notifying.
If any one of these four components stop working for some reason, the others can continue their processing. If any one or more of these queues end up with a large number of messages to be processed, additional individual components can be added to help improve performance. There is another added benefit to this architecture. Since the system is now comprised of independent components, the team that is building and maintaining the system can work on and maintain these components separately.
The efficiencies of decoupling are not only realized at the hardware and software level. They can also improve development process and throughput. Decoupling Your Components: Lesson #1. Use SQS as the communication glue to bind independent components together into a fail-safe, scalable, and high performant cohesive application.
- Benefits of cloud services
- Making architectures scalable
- Examining cloud constraints
- Virtual servers, EC2, and Elastic IP
- Using the Amazon machine image
- Elastic load balancing
- Using CloudWatch for monitoring
- Security Models
- Elastic block storage
- S3, CloudFront, and Elastic Beanstalk
- Handling queues, workflows, and notifications
- Caching options and services
- Identity and access management
- Creating a custom server image
- Application deployment strategies
- Serverless architectures