How to use Amazon Lambda with Amazon SQS?

AWSMAG-Using Amazon SQS with Amazon Lambda
Using Amazon SQS with Amazon Lambda

Amazon SQS (Simple Queue Service) is a fully managed queue service that allows you to de-couple and scale for applications. Amazon SQS offers two types of queues: Standard and FIFO (First In First Out) queues.

You can read more about the Amazon SQS here. We also have a collection of articles related to Amazon SQS.

In this article, we will create a Lambda function to consume the messages in the SQS. The deployment will be using an AWS SAM template and we will also create a DLQ (Dead Letter Queue) to move the failed messages to it.

How does this work?

Using Amazon SQS with Amazon Lambda
Using Amazon SQS with Amazon Lambda

When an application sends a message to the SQS, the lambda poling service polls the queue and invokes your Lambda function synchronously. The Lambda function receives an event that contains queue messages. Lambda Polling service reads messages in batches and invokes your function once for each batch. Each batch can contain up to 10 messages. Once the processing is complete by the function, the Lambda will delete messages from the SQS. If a message fails it will be retained in the SQS and we can retry the message until the re-drive policy is triggered.

An example of an event with 1 message will be something like this.

{
    "Records": [
        {
            "messageId": "2e1424d4-87a3-44ab-83d2-9c92662be6da",
            "receiptHandle": "AQEBwJnKyrHigUM+/7q1rGgNqicHq...",
            "body": "Example message.",
            "attributes": {
                "ApproximateReceiveCount": "1",
                "SentTimestamp": "1648534568000",
                "SenderId": "AIDAIENQZJOLO2QZJOLOVO",
                "ApproximateFirstReceiveTimestamp": "1648534568000"
            },
            "messageAttributes": {},
            "md5OfBody": "c2f3c80ed14c967acaaf3b09a2e4f268",
            "eventSource": "aws:sqs",
            "eventSourceARN": "arn:aws:sqs:eu-west-2:1648534568000:example-queue",
            "awsRegion": "eu-west-2"
        }
    ]
}
Example of an Event from SQS to Lambda

How to configure SQS to use with Lambda?

Following is an example of an AWS SAM template that will deploy a DLQ and a primary SQS along with the Lambda function to process the messages. The example shows the FIFO SQS, but the concept is similar to the standard SQS.

πŸ’‘
One point to note here is that a FIFO SQS will always have a FIFO DLQ. 
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description:
  Node.js barebone for AWS Lambda
Globals:
  Function:
    Timeout: 300
    Runtime: nodejs14.x
    MemorySize: 512
Resources:
  MainDLQSQS:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: MainsDLQ.fifo
      ContentBasedDeduplication: true
      FifoQueue: true
      MessageRetentionPeriod: 864000
  MainSQS:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: Mains.fifo
      ContentBasedDeduplication: true
      FifoQueue: true
      MessageRetentionPeriod: 864000
      VisibilityTimeout: 180
      RedrivePolicy:
        deadLetterTargetArn: !GetAtt MainDLQSQS.Arn
        maxReceiveCount: 5
  MainFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: MainProcessorFunction
      CodeUri: "src"
      Handler: index.handleEvent
      ReservedConcurrentExecutions: 5
      Events:
        SQSEvent:
          Type: SQS
          Properties:
            Queue: !GetAtt MainSQS.Arn
            BatchSize: 10
      Policies:
        - SQSPollerPolicy:
            QueueName: !GetAtt MainSQS.QueueName
AWS SAM template for using AWS SQS with Amazon Lambda
πŸ“š
If you are not familiar with AWS SAM, you can have a look at the following articles:
- What is AWS SAM?
- Getting Started with AWS SAM
- Understanding AWS SAM Template

If you look at the above SAM Template, we have defined three resources.

  • MainDLQSQS: This is a dead letter queue and will act as a secondary SQS to hold the messages which will fail to process properly.
  • MainSQS: This is the main SQS. All the messages from the publisher will land in this SQS. We will try to process the message up to 5 times. If we fail to process it properly and were not able to delete it from the SQS, the message will be moved to DLQ. This is defined using the re-drive policy defined in the template.
  • MainFunction: This is our Lambda function defined to receive the message. The event is defined to link to SQS and will receive the message in the batch of 10. The policy gives access to poll the SQS and invokes a Lambda function when it is ready.
πŸ’»
You can access the template and a sample repository here.

Conclusion:

SQS is a great service that can help you in developing some async serverless applications. FIFO SQS can help you maintain order while processing the messages in case the order is something that is very important in your application.

Subscribe to AWSMAG

Don’t miss out on the latest issues. Sign up now to get access to the library of members-only issues.
jamie@example.com
Subscribe