Amazon SQS (Simple Queue Service) is a fully managed queue service that allows you to de-couple and scale for applications. Amazon SQS offers two types of queues: Standard and FIFO (First In First Out) queues.
In this article, we will create a Lambda function to consume the messages in the SQS. The deployment will be using an AWS SAM template and we will also create a DLQ (Dead Letter Queue) to move the failed messages to it.
How does this work?
When an application sends a message to the SQS, the lambda poling service polls the queue and invokes your Lambda function synchronously. The Lambda function receives an event that contains queue messages. Lambda Polling service reads messages in batches and invokes your function once for each batch. Each batch can contain up to 10 messages. Once the processing is complete by the function, the Lambda will delete messages from the SQS. If a message fails it will be retained in the SQS and we can retry the message until the re-drive policy is triggered.
An example of an event with 1 message will be something like this.
How to configure SQS to use with Lambda?
Following is an example of an AWS SAM template that will deploy a DLQ and a primary SQS along with the Lambda function to process the messages. The example shows the FIFO SQS, but the concept is similar to the standard SQS.
If you look at the above SAM Template, we have defined three resources.
- MainDLQSQS: This is a dead letter queue and will act as a secondary SQS to hold the messages which will fail to process properly.
- MainSQS: This is the main SQS. All the messages from the publisher will land in this SQS. We will try to process the message up to 5 times. If we fail to process it properly and were not able to delete it from the SQS, the message will be moved to DLQ. This is defined using the re-drive policy defined in the template.
- MainFunction: This is our Lambda function defined to receive the message. The event is defined to link to SQS and will receive the message in the batch of 10. The policy gives access to poll the SQS and invokes a Lambda function when it is ready.
SQS is a great service that can help you in developing some async serverless applications. FIFO SQS can help you maintain order while processing the messages in case the order is something that is very important in your application.