AWS SNS

SNS messages contain the notification content in the "data" property. See Notification types v1 for more information on this content.

Setting up AWS SNS topic for notifications

To enable the delivery of Agile Data Engine notification events to AWS SNS, follow these steps:

1. Create an SNS topic with following settings:

• Type:

  • FIFO if you want strictly-preserved message ordering and exactly-once message delivery.

  • Standard if you prefer best-effort message ordering and at-least-once message delivery.

• Subscription protocols: SQS for both types. Standard also supports SQS, Lambda, HTTP, SMS, email, mobile application endpoints

• Name: Choose a meaningful name for your topic, e.g. AdeNotificationEventsDev

• Other optional parameters: Refer to the AWS documentation on https://docs.aws.amazon.com/sns/latest/dg/sns-create-topic.html

2. (Optional) Configure subscriptions

Connect and configure subscriptions to your SNS topic to deliver events to other services, or you can do this later.

3. Create an IAM role

This IAM role should be assumable by the ADE AWS account, which will send messages to your SNS topic. It should also contain an external ID for unique association with the role.

The ADE AWS account ID and external ID will be provided in step 5.

More information on https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_common-scenarios_third-party.html

Follow these steps to create the IAM role:

• Name: Choose a meaningful name for your IAM role, e.g.: AdeNotificationEventsThirdPartyRole

• Permissions: Define permissions to publish to the SNS topic, using a policy like the one below:

  CODE

  {
    "Statement":[{
      "Sid":"AllowPublishToMyTopic",
      "Effect":"Allow",
      "Action":"sns:Publish",
      "Resource":"arn:aws:sns:eu-west-1:123456789012:AdeNotificationEventsDev"
    }]
  }  

  Use the ARN from your previously created topic in this IAM role policy.

4. Contact ADE support

Provide the following data to Agile Data Engine support:

• Topic ARN

• IAM role ARN

5. Receive ADE AWS account ID and external ID

ADE support will configure their side and respond to you with the ADE AWS account ID and external ID (refer to Step 3).

Update the trusted relationship in your IAM role:

• Assumable by AWS account:  <Received ADE AWS account ID>

• Require an external ID: <Received external ID>

6. Inform ADE support

Inform ADE support when the trusted relationship has been changed so that a test notification event can be sent out.

SNS Record example

{
  "Message": {
    "sourceId": "cf1e80db-3c1a-4b7b-88b7-2518b229603a",
    "tenant": "s1234567",
    "installation": "datahub",
    "environment": "runtime",
    "notificationId": "4fd01f17-4403-4a20-be86-3728611b4ad4",
    "notificationChannel": "0bf43704-8ac7-4ca8-86e1-155de3e56389",
    "notificationLevel": "WARN|ERROR",
    "notificationType": "OPERATIONAL",
    "notificationTime": "2023-01-01T12:50:05.358Z",
    "notificationSource": "AdeCore/WorkflowOrchestration",
    "contentText": "Workflow MY_DAG failed: Some problem description.",
    "contentType": "WorkflowFailure",
    "contentLink": "https://external.runtime.datahub.s1234567.agiledataengine.com/dagger/graph?dag_id=MY_DAG&execution_date=2023-01-01T10%3A16%3A58.119141%2B00%3A00"
  },
  "MessageAttributes": {
    "dataVersion": "1",
    "subject": "/s1234567/datahub/runtime",
    "eventTime": "2023-01-01T12:50:05.358Z",
    "eventType": "AdeCore.WorkflowOrchestration.WorkflowFailure",
    "id": "4fd01f17-4403-4a20-be86-3728611b4ad4"
  }
}