AWS SNS/SQS

To setup AWS SNS/SQS for pub/sub, create a component of type pubsub.snssqs. .

Warning

The above example uses secrets as plain strings. It is recommended to use [a secret store for the secrets]]().

Conforming with AWS specifications

Dapr created SNS topic and SQS queue names conform with . By default, Dapr creates an SQS queue name based on the consumer app-id, therefore Dapr might perform name standardization to meet with AWS specifications.

SNS/SQS component behavior

When the pub/sub SNS/SQS component provisions SNS topics, the SQS queues and the subscription behave differently in situations where the component is operating on behalf of a message producer (with no subscriber app deployed), than in situations where a subscriber app is present (with no publisher deployed).

Due to how SNS works without SQS subscription in publisher only setup, the SQS queues and the subscription behave as a “classic” pub/sub system that relies on subscribers listening to topic messages. Without those subscribers, messages:

  • Cannot be passed onwards and are effectively dropped
  • Are not available for future subscribers (no replay of message when the subscriber finally subscribes)

SQS FIFO

Using SQS FIFO (fifo metadata field set to "true") per AWS specifications provides message ordering and deduplication, but incurs a lower SQS processing throughput, among other caveats.

To avoid losing the order of messages delivered to consumers, the FIFO configuration for the SQS Component requires the concurrencyMode metadata field set to "single".

Default parallel concurrencyMode

Since v1.8.0, the component supports the "parallel" concurrencyMode as its default mode. In prior versions, the component default behavior was calling the subscriber a single message at a time and waiting for its response.

SQS dead-letter Queues

When configuring the PubSub component with SQS dead-letter queues, the metadata fields messageReceiveLimit and sqsDeadLettersQueueName must both be set to a value. For messageReceiveLimit, the value must be greater than 0 and the sqsDeadLettersQueueName must not be empty string.

Important

When running the Dapr sidecar (daprd) with your application on EKS (AWS Kubernetes) node/pod already attached to an IAM policy defining access to AWS resources, you must not provide AWS access-key, secret-key, and tokens in the definition of the component spec.

For local development, the is used to integrate AWS SNS/SQS. Follow these instructions to run localstack.

To run localstack locally from the command line using Docker, apply the following cmd:

  1. docker run --rm -it -p 4566:4566 -p 4571:4571 -e SERVICES="sts,sns,sqs" -e AWS_DEFAULT_REGION="us-east-1" localstack/localstack

See for information about authentication-related attributes.

To run localstack on Kubernetes, you can apply the configuration below. Localstack is then reachable at the DNS name http://localstack.default.svc.cluster.local:4566 (assuming this was applied to the default namespace), which should be used as the endpoint.

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: localstack
  5. spec:
  6.   # using the selector, we will expose the running deployments
  7.   # this is how Kubernetes knows, that a given service belongs to a deployment
  8.   selector:
  9.     matchLabels:
  10.       app: localstack
  11.   template:
  12.     metadata:
  13.       labels:
  14.         app: localstack
  15.     spec:
  16.       containers:
  17.       - name: localstack
  18.         image: localstack/localstack:latest
  19.         ports:
  20.           # Expose the edge endpoint
  21.           - containerPort: 4566
  23. kind: Service
  24. apiVersion: v1
  25. metadata:
  26.   name: localstack
  27.   labels:
  28.     app: localstack
  29. spec:
  30.   selector:
  31.     app: localstack
  32.   ports:
  33.   - protocol: TCP
  34.     port: 4566
  35.     targetPort: 4566
  36.   type: LoadBalancer

In order to run in AWS, create or assign an IAM user with permissions to the SNS and SQS services, with a policy like:

Plug the AWS account ID and AWS account secret into the accessKey and secretKey in the component metadata, using Kubernetes secrets and secretKeyRef.

Alternatively, let’s say you want to provision the SNS and SQS assets using your own tool of choice (e.g. Terraform) while preventing Dapr from doing so dynamically. You need to enable disableEntityManagement and assign your Dapr-using application with an IAM Role, with a policy like:

  1. {
  2.   "Version": "2012-10-17",
  3.     {
  4.       "Sid": "YOUR_POLICY_NAME",
  5.       "Effect": "Allow",
  6.       "Action": [
  7.         "sqs:DeleteMessage",
  8.         "sqs:ReceiveMessage",
  9.         "sqs:ChangeMessageVisibility",
  10.         "sqs:GetQueueUrl",
  11.         "sqs:GetQueueAttributes",
  12.         "sns:Publish",
  13.         "sns:ListSubscriptionsByTopic",
  14.         "sns:GetTopicAttributes"
  16.       ],
  17.       "Resource": [
  18.         "arn:aws:sns:AWS_REGION:AWS_ACCOUNT_ID:APP_TOPIC_NAME",
  19.         "arn:aws:sqs:AWS_REGION:AWS_ACCOUNT_ID:APP_ID"
  20.       ]
  21.     }
  23. }

In the above example, you are running your applications on an EKS cluster with dynamic assets creation (the default Dapr behavior).