[Bug]: SNS -> SQS Subscription wrong ARN when working with localstack- Invalid parameter: SQS endpoint ARN

[MichalOleszczuk]

Bug description

autoSubscribe not working for resource with Protocol: 'sqs',

Environment

• Windows 10
• Node version: 16.20.0
• NPM version: 8.19.4
• "serverless": "3.27.0",
• "serverless-esbuild": "1.39.0",
• "serverless-offline": "12.0.4",
• "serverless-offline-sns": "0.76.0",
• "serverless-offline-sqs": "7.3.2",

Bugfix or workaround

Make autoSubscribe feature work by providing resource arn in Endpoint: subscribeEndpoint, for Protocol: 'sqs' not http url.

Details

We can make autoSubscribe work for SQS subscriptions with sns-endpoint config option by allowing to pull queue resource ARN.
Currently when code reaches this.sns.subscribe(params, (err, data) => ... then params object is already wrong. When we provide resource with Protocol: 'sqs', line Endpoint: subscribeEndpoint, should not have URL in it. Since mixing protocol different then 'http' and url provided in endpoint will result in error from AW SDK
⚠️ Invalid parameter: SQS endpoint ARN.
And this makes sense since when using Protocol: 'SQS' SDK is expecting ARN not http url.

For serverless.ts configuration:

      'serverless-offline-sns': {
        'sns-endpoint': 'http://localhost:4566',
        accountId: process.env.AWS_ACCOUNT_ID,
        debug: true,
      },
    [...]
  resources: {
    Resources: {
      SnsSynchronizeProfileEmailTopic: {
        Type: 'AWS::SNS::Topic',
        Properties: {
          TopicName: 'synchronize-profile-email-topic',
        },
      },
      SnsSynchronizeProfileEmailSubscription: {
        Type: 'AWS::SNS::Subscription',
        Properties: {
          Protocol: 'sqs',
          Endpoint: { 'Fn::GetAtt': ['SynchronizeProfileEmailQueue', 'Arn'] },
          TopicArn: { Ref: 'SnsSynchronizeProfileEmailTopic' },
          RawMessageDelivery: 'true',
        },
      },
      SynchronizeProfileEmailQueue: {
        Type: 'AWS::SQS::Queue',
        Properties: {
          QueueName: 'synchronize-profile-email',
        },
      },
    },
  },

This code fails

when this code works

And it triggers lambda configured like below in the end

  functions: {
    [config.name]: {
      handler: relativePathToHandler,
      events: [
        {
          sqs: {
            enabled: true,
            batchSize: 1,
            maximumBatchingWindow: 1,
            arn: { 'Fn::GetAtt': ['SynchronizeProfileEmailQueue', 'Arn'] },
          },
        },
      ],
    },
  },

script sending message to SNS

[MichalOleszczuk]

@mj1618 - if no one has ready fix for it in spare time I will propose PR to fox this issue. Will npm publish work on successful PR?

[pikos-apikos]

I have the same issue when using docker
Here is my docker compose file

version: '3.0'

services:
  localstack:
    image: localstack/localstack:latest
    environment:
      - SERVICES=sqs,sns
      - AWS_DEFAULT_REGION=eu-west-2
      - EDGE_PORT=4566
    ports:
      - '4566-4597:4566-4597'
    volumes:
      - "${TMPDIR:-/tmp/localstack}:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"

  setup-resources:
    image: mesosphere/aws-cli
    volumes:
      - ./dev_env:/project/dev_env
    environment:
      - AWS_ACCESS_KEY_ID=AKIAEXAMPLE123
      - AWS_SECRET_ACCESS_KEY=AWSSECRETACCESSEY123
      - AWS_DEFAULT_REGION=eu-west-2
    entrypoint: /bin/sh -c
    command: >
      "
        sleep 20
        # Executing SNS
        aws sns create-topic --name local_sns --endpoint-url=http://localstack:4566 
        # Executing SQS
        aws sqs create-queue --endpoint-url=http://localstack:4566 --queue-name local_queue;
        # Subscribing to SNS to SQS
        aws --endpoint-url=http://localstack:4566 sns subscribe --topic-arn arn:aws:sns:eu-west-2:000000000000:local_sns --protocol sqs --notification-endpoint http://localstack:4566/queue/local_queue
      "
    depends_on:
      - localstack

[MichalOleszczuk]

@pikos-apikos - for now you can workaround the issue with commenting out your sns subscription

      // INFO: this has to be commented until https://github.com/mj1618/serverless-offline-sns/issues/180. Not working on serverless-offline
      // SnsTestSubscription: {
      //   Type: 'AWS::SNS::Subscription',
      //   Properties: {
      //     Protocol: 'sqs',
      //     Endpoint: { 'Fn::GetAtt': ['TestQueue', 'Arn'] },
      //     TopicArn: { Ref: 'SnsTestTopic' },
      //     RawMessageDelivery: 'true',
      //   },
      // },

And adding subscriptions in your serverless -> custom -> serverless-offline-sns
Look for "subscriptions"

subscriptions: [
          {
            topic: {
              topicName: 'test-topic',
            },
            queue: `arn:aws:sqs:${process.env.AWS_DEFAULT_REGION}:${process.env.AWS_ACCOUNT_ID}:test-queue`,
          },
        ]

My sns-offline setup

      'serverless-offline-sns': {
        port: 4002,
        'sns-endpoint': 'http://localhost:4566',
        accountId: process.env.AWS_ACCOUNT_ID,
        autoSubscribe: false,
        subscriptions: [
          {
            topic: {
              topicName: 'test-topic',
            },
            queue: `arn:aws:sqs:${process.env.AWS_DEFAULT_REGION}:${process.env.AWS_ACCOUNT_ID}:test-queue`,
          },
        ],
      },

[VirtueMe]

Fixed in Phase A batch release (PR #249).