How do you choose between Events, Queues, Topics, and Streams for your Serverless Application?

Leave a Comment / Uncategorized / By

When you are building microservices-based serverless applications, one of the important things will be communication between the services.  When we talk about serverless in AWS, the first thing that comes to mind is the AWS Lambda service provided by AWS. In this post we will focus on how do you choose what to use as an invocation mechanism for AWS Lambda.

When a Lambda function is invoked, it receives the event from one of the sources and performs the task. Four important message patterns for serverless are:-

  • Events: using Event buses provided by Amazon Event Bridge.
  • Queues: be it LIFO(last in first out) or FIFO(first in first out) provided by Amazon SQS.
  • Topics: Publish/Subscribe model provided by Amazon SNS.
  • Streams: a real-time stream of messages which can invoke Lambda. This is provided by Amazon Kinesis or Dynamodb Streams.
💡
Before we move ahead with this article, the important thing you should know is the Lambda Execution models. If you are not aware of these, you can read about them here.

Amazon EventBridge (Events)

Amazon EventBridge is a serverless event bus service for SaaS and AWS services. It is fully managed and you only pay for what you use. There are some native integrations available with some of the SAAS providers. You can use Amazon EventBridge to easily build event-driven architectures and workflows for your serverless applications.

Use Cases for Amazon EventBridge:

  • When you want to take action based on the actions in any other AWS service. One example can be a file uploaded to S3. The S3 can send a notification to EventBridge. This will invoke a Lambda function to process the notification accordingly.
  • You can also run some workflows if required. For example, a user is onboarded to your application and now needs some tasks to be completed in a proper sequence. Your application can send an event to Amazon EventBridge which can trigger a step function to process a workflow.
  • As mentioned above, Amazon EventBridge has native support for other SAAS applications. An example is MongoDB Atlas. The change streams in MongoDB can be sent to an EventBridge and the event can be used to take further action by calling a Lambda or a step function.

You can use EventBridge:

  • When the communication is 1 to Many. There is a limitation of 5 targets per rule.
  • When you need native integration with the other SaaS Platforms supported by Amazon EventBridge

Amazon SNS (Topics)

Amazon SNS (Simple Notification Service) is a fully managed messaging service. The topics are the first-class citizen of Amazon SNS. Amazon SNS is used to implement pub/sub (publication/subscription) methodology to work with the messages passed through this service. The messages are published to a topic and the same message is fan-out to multiple interested subscribers.

You can read more about the Amazon SNS here. We also have a collection of articles related to Amazon SNS.

You can use SNS:

  • When you need a 1 to Many Fan-out processing. Many subscribers can subscribe to a topic and receives the same message when it is published the topic.
  • Amazon SNS has a high throughput and supports the passing of a lot of messages to the subscribers of the topic.

Amazon SQS (Queues)

Amazon SQS (Simple Queue Service) is a fully managed queue service that allows you to de-couple and scale for applications. Amazon SQS offers two types of queue: Standard and FIFO (First In First Out) queue.

You can read more about the Amazon SQS here. We also have a collection of articles related to Amazon SQS.

You can use SQS:

  • When you need a reliable 1-1 asynchronous communication.
  • When you want to hold the messages in a pool before processing them. SQS allows you to hold the messages for up to 14 days. You can also implement a retry mechanism where if the message fails to process in a certain amount of retires, the message will be pushed to a separate SQS i.e. DLQ (Dead Letter Queue) for further actions.
  • One of the features provided by SQS is the FIFO Queue. The FIFO queue allows you to process the messages in a particular order. This is a good use case when you want to make sure the messages received from a system should be processed in a specific order.

Amazon Kinesis Streams (Streams)

Amazon Kinesis is a fully managed service for collecting and processing real-time data streams. A stream can be divided into shards with a fixed amount of capacity and throughput. A Lambda service will invoke the Lambda function and sends the batches of messages for processing. Many consumers can be attached to the same stream and can receive the messages at the same time.

You can use Kinesis Streams:

  • When you need real-time processing of data. The data in a  stream is collected and processed as soon as it is collected. One example can be a change in the data stored in a database that supports streams.
  • When you need to send the same stream of data for the different use cases. The log data can be streamed to run the real-time analysis of the application\’s security state and can also be streamed to be saved in an S3 for processing after some time.

Conclusion:

This article gives an introduction to the four methods and when to use each method in your serverless application. This does not mean that you cannot use the above-mentioned services for any other use case. This is just a guide and you should choose a service/ method based on your use case. You can also combine two services in orderto achieve a better solution for your scenario. One example can be to use SNS to fan out an event to multiple SQS and then do specific processing based on the message in the SQS.

Related Posts