Google Pub/Sub

Setting up Google Pub/Sub topic for notifications

To enable the delivery of Agile Data Engine notification events to Google Cloud Pub/Sub, follow these steps:

1. Create a Pub/Sub topic

Create a Pub/Sub topic that will receive notification messages.

• Console:
  Go to Google Cloud Console – Pub/Sub
  Click “Create Topic”

• Settings:

  • Topic ID: Choose a descriptive name, e.g., ade-notification-events-dev

  • Message Storage: Default options are usually fine unless you need specific retention settings.

• Command line (gcloud):

  CODE

  gcloud pubsub topics create ade-notification-events-dev

Refer to Google documentation on creating topics for advanced configurations.

2. Create a dedicated IAM User / Service account

Set up a service account that ADE will use to publish messages to your Pub/Sub topic.

Recommended name:
ade-notification-events-pubsub-publisher

Permissions:
Grant the roles/pubsub.publisher role on the specific topic.

2.1 Option A: Using the Console

1. Go to IAM & Admin > Service Accounts.

2. Click "Create Service Account".

3. Enter the name and description.

4. Assign the Pub/Sub Publisher role scoped to your topic.

2.2 Option B: Using gcloud

gcloud iam service-accounts create ade-notification-events-pubsub-publisher \
  --description="ADE Pub/Sub publisher account" \
  --display-name="ADE PubSub Publisher"

gcloud pubsub topics add-iam-policy-binding ade-notification-events-dev \
  --member="serviceAccount:ade-notification-events-pubsub-publisher@<your-project-id>.iam.gserviceaccount.com" \
  --role="roles/pubsub.publisher"

3. Extract Credentials

Choose one of the following methods to authenticate with your service account.

3.1 Service Account JSON Key

Generate a key file to securely share with ADE:

1. Go to IAM & Admin > Service Accounts.

2. Select the service account: ade-notification-events-pubsub-publisher.

3. Open the "Keys" tab.

4. Click "Add Key" > "Create new key".

5. Choose JSON, then download and store the file securely.

Important: This JSON key provides publishing access to your topic. Treat it as a sensitive secret. Do not send via plaintext or email.

3.2 Workload Identity Federation (WIF)

To use WIF instead of a key file:

1. Add a new AWS provider to a new or existing WIF pool.
   Suggested name: aws-ade-notification-events-provider

2. ADE Support will provide the AWS Account ID (see Step 4).

3. In Attribute Mapping, set:

google.subject = assertion.arn.split(":")[4] + ":" + assertion.arn.extract("assumed-role/{role}/")

3.2.1 Service Account Impersonation

1. Grant WIF impersonation access to the ade-notification-events-pubsub-publisher service account.

2. Set subject as the attribute name, with the value provided by ADE Support.

3. Download the Client Library Configuration and securely share it with ADE Support (see Step 4).

4. Contact ADE Support

Send the following details to Agile Data Engine (via a secure channel):

• Google Cloud Project ID

• Pub/Sub Topic ID (e.g., ade-notification-events-dev)

• Credentials (either the JSON key or WIF config)

  • If using WIF, inform ADE so they can provide the AWS Account ID and subject value for the attribute mapping.

ADE will finalize the integration setup on their side using this information.

5. Receive test messages from ADE

Once ADE has confirmed the integration, you should begin receiving notification events in your topic. You can verify delivery by:

• Subscribing to the topic (e.g., using a pull subscription or forwarding to another system)

• Inspecting message logs via the Google Cloud Console

Pub/Sub Message example

ADE messages will follow a structure similar to this:

[
  {
    "attributes": {
      "id": "4fd01f17-4403-4a20-be86-3728611b4ad4",
      "eventTime": "2022-01-13T12:50:05.358Z",
      "eventType": "AdeCore.WorkflowOrchestration.WorkflowFailure",
      "subject": "/s1234567/datahub/runtime",
      "dataVersion": 1
    },
    "data": {
      "sourceId": "cf1e80db-3c1a-4b7b-88b7-2518b229603a",
      "tenant": "s1234567",
      "installation": "datahub",
      "environment": "runtime",
      "notificationId": "4fd01f17-4403-4a20-be86-3728611b4ad4",
      "notificationChannel": "0bf43704-8ac7-4ca8-86e1-155de3e56389",
      "notificationLevel": "WARN|ERROR",
      "notificationType": "OPERATIONAL",
      "notificationTime": "2022-01-13T12:50:05.358Z",
      "notificationSource": "AdeCore/WorkflowOrchestration",
      "contentText": "Workflow MY_DAG failed: Some problem description.",
      "contentType": "WorkflowFailure",
      "contentLink": "https://external.runtime.datahub.s1234567.agiledataengine.com/dagger/graph?dag_id=MY_DAG&execution_date=2022-01-12T10%3A16%3A58.119141%2B00%3A00"
    }
  }
]