How to use Amazon Lambda with Amazon SQS?

Leave a Comment / Uncategorized / By

Amazon SQS (Simple Queue Service) is a fully managed queue service that allows you to de-couple and scale for applications. Amazon SQS offers two types of queues: Standard and FIFO (First In First Out) queues.

You can read more about the Amazon SQS here. We also have a collection of articles related to Amazon SQS.

In this article, we will create a Lambda function to consume the messages in the SQS. The deployment will be using an AWS SAM template and we will also create a DLQ (Dead Letter Queue) to move the failed messages to it.

How does this work?

\"Using
Using Amazon SQS with Amazon Lambda

When an application sends a message to the SQS, the lambda poling service polls the queue and invokes your Lambda function synchronously. The Lambda function receives an event that contains queue messages. Lambda Polling service reads messages in batches and invokes your function once for each batch. Each batch can contain up to 10 messages. Once the processing is complete by the function, the Lambda will delete messages from the SQS. If a message fails it will be retained in the SQS and we can retry the message until the re-drive policy is triggered.

An example of an event with 1 message will be something like this.

{    \"Records\": [        {            \"messageId\": \"2e1424d4-87a3-44ab-83d2-9c92662be6da\",            \"receiptHandle\": \"AQEBwJnKyrHigUM+/7q1rGgNqicHq...\",            \"body\": \"Example message.\",            \"attributes\": {                \"ApproximateReceiveCount\": \"1\",                \"SentTimestamp\": \"1648534568000\",                \"SenderId\": \"AIDAIENQZJOLO2QZJOLOVO\",                \"ApproximateFirstReceiveTimestamp\": \"1648534568000\"            },            \"messageAttributes\": {},            \"md5OfBody\": \"c2f3c80ed14c967acaaf3b09a2e4f268\",            \"eventSource\": \"aws:sqs\",            \"eventSourceARN\": \"arn:aws:sqs:eu-west-2:1648534568000:example-queue\",            \"awsRegion\": \"eu-west-2\"        }    ]}
Example of an Event from SQS to Lambda

How to configure SQS to use with Lambda?

Following is an example of an AWS SAM template that will deploy a DLQ and a primary SQS along with the Lambda function to process the messages. The example shows the FIFO SQS, but the concept is similar to the standard SQS.

💡
One point to note here is that a FIFO SQS will always have a FIFO DLQ. 
AWSTemplateFormatVersion: \"2010-09-09\"Transform: AWS::Serverless-2016-10-31Description:  Node.js barebone for AWS LambdaGlobals:  Function:    Timeout: 300    Runtime: nodejs14.x    MemorySize: 512Resources:  MainDLQSQS:    Type: AWS::SQS::Queue    Properties:      QueueName: MainsDLQ.fifo      ContentBasedDeduplication: true      FifoQueue: true      MessageRetentionPeriod: 864000  MainSQS:    Type: AWS::SQS::Queue    Properties:      QueueName: Mains.fifo      ContentBasedDeduplication: true      FifoQueue: true      MessageRetentionPeriod: 864000      VisibilityTimeout: 180      RedrivePolicy:        deadLetterTargetArn: !GetAtt MainDLQSQS.Arn        maxReceiveCount: 5  MainFunction:    Type: AWS::Serverless::Function    Properties:      FunctionName: MainProcessorFunction      CodeUri: \"src\"      Handler: index.handleEvent      ReservedConcurrentExecutions: 5      Events:        SQSEvent:          Type: SQS          Properties:            Queue: !GetAtt MainSQS.Arn            BatchSize: 10      Policies:        - SQSPollerPolicy:            QueueName: !GetAtt MainSQS.QueueName
AWS SAM template for using AWS SQS with Amazon Lambda
📚
If you are not familiar with AWS SAM, you can have a look at the following articles:
– What is AWS SAM?
– Getting Started with AWS SAM
– Understanding AWS SAM Template

If you look at the above SAM Template, we have defined three resources.

  • MainDLQSQS: This is a dead letter queue and will act as a secondary SQS to hold the messages which will fail to process properly.
  • MainSQS: This is the main SQS. All the messages from the publisher will land in this SQS. We will try to process the message up to 5 times. If we fail to process it properly and were not able to delete it from the SQS, the message will be moved to DLQ. This is defined using the re-drive policy defined in the template.
  • MainFunction: This is our Lambda function defined to receive the message. The event is defined to link to SQS and will receive the message in the batch of 10. The policy gives access to poll the SQS and invokes a Lambda function when it is ready.
💻
You can access the template and a sample repository here.

Conclusion:

SQS is a great service that can help you in developing some async serverless applications. FIFO SQS can help you maintain order while processing the messages in case the order is something that is very important in your application.

Related Posts