From the course: AWS for Developers: SNS, SQS, and SWF

Messaging queues - Amazon Web Services (AWS) Tutorial

From the course: AWS for Developers: SNS, SQS, and SWF

Start my 1-month free trial

Messaging queues

“

- When your application has a lot of processing to do, such as our resume parser, it's best to queue up these tasks in the background and let your users go on to do something else. A queue can also help relieve some of the pressure for your database if you have to ingest a lot of data or do a lot of processing on that data before it's written to its final destination. You can also use a queue as a messaging provider. And it can be a way to pass information around in your system and to alert your other components when certain events have happened. Now, even if you're familiar with modern day databases, I recommend you read the book "Designing Data-Intensive Applications." It doesn't favor one specific database engine, but it talks about the broad strokes and how the database technologies differ from each other, and when is the right time to employ each strategy. The final section of the book discusses messaging queues, and it can give you some insights on when it's appropriate to deploy these into an application, but let me try to summarize it for you. Think of a messaging queue like a postal mailbox. Imagine that someone puts a letter into your mailbox. You then come by later and pull the message out. Now, the mail carrier is the producer, and they add items to your queue, or your mailbox. You come by at a later time. You pull the messages out, so you are the consumer of this queue. But with a messaging queue you can also have multiple producers and multiple consumers. One way to think about this is like a bunch of short order cooks at a diner. Someone on the wait staff puts an order ticket up on the line. They are the producers. One of the cooks pulls the ticket on the right, which is the oldest ticket, and starts making that order. Another cook will come by and then pull the next ticket, but because the cooks pull the order ticket off the line, you won't have two cooks working on the same order and doing duplicate work. Right now in our example application the resume parser runs in line when the resume is uploaded, tying up the server. Now, you could just create a cron job that runs the resume parser and just have it look in the database to see if there's work to do. But only one copy of the resume parser can ever run at one time, because it can't coordinate work with a second or third parser. Using a messaging queue like Amazon SQS will allow us to have several copies of the resume parser running at the same time. And SQS will split up the work between the different parsers so we don't have two parsers working against the same resume. Let's set up a test server, and then we can model how this will work.

Contents