Google Pub/Sub

Setting up Google Pub/Sub topic for notifications

To enable the delivery of Agile Data Engine notification events to Google Cloud Pub/Sub, follow these steps:

1. Create a Pub/Sub topic

Create a Pub/Sub topic that will receive notification messages.

Console:
Go to Google Cloud Console – Pub/Sub
Click “Create Topic”
Settings:
- Topic ID: Choose a descriptive name, e.g., ade-notification-events-dev
- Message Storage: Default options are usually fine unless you need specific retention settings.

Command line (gcloud):

CODE

gcloud pubsub topics create ade-notification-events-dev

Refer to Google documentation on creating topics for advanced configurations.

2. Create a dedicated IAM user/service account

Create a service account that ADE can use to publish messages to your Pub/Sub topic.

Name: Choose a meaningful name, e.g., ade-pubsub-publisher
Permissions:
- Grant the roles/pubsub.publisher role on the specific topic.

Steps:

Go to IAM & Admin > Service Accounts
Click “Create Service Account”
Assign the Pub/Sub Publisher role scoped to your topic

Alternatively, use gcloud:

CODE

gcloud iam service-accounts create ade-pubsub-publisher \
  --description="ADE Pub/Sub publisher account" \
  --display-name="ADE PubSub Publisher"
gcloud pubsub topics add-iam-policy-binding ade-notification-events-dev \
  --member="serviceAccount:ade-pubsub-publisher@<your-project-id>.iam.gserviceaccount.com" \
  --role="roles/pubsub.publisher"

3. Extract credentials (Service Account JSON key)

Generate a key file to be shared securely with ADE.

Go to IAM & Admin > Service Accounts
Select your service account (ade-pubsub-publisher)
Open "Keys" tab
Click “Add Key” → “Create new key”
- Choose JSON
- Save the key file securely (do not send it in plain text or over email)

⚠️ Important: This JSON key grants publishing access to your Pub/Sub topic. Treat it as a secret.

4. Contact ADE Support

Provide the following information to Agile Data Engine support:

Google Cloud Project ID
Pub/Sub Topic ID (e.g., ade-notification-events-dev)
Service Account JSON key (through secured channel, not plain text)

ADE will configure the integration on their side using these credentials.

5. Receive test messages from ADE

Once ADE has confirmed the integration, you should begin receiving notification events in your topic. You can verify delivery by:

Subscribing to the topic (e.g., using a pull subscription or forwarding to another system)
Inspecting message logs via the Google Cloud Console

Pub/Sub Message example

ADE messages will follow a structure similar to this:

CODE

[
  {
    "attributes": {
      "id": "4fd01f17-4403-4a20-be86-3728611b4ad4",
      "eventTime": "2022-01-13T12:50:05.358Z",
      "eventType": "AdeCore.WorkflowOrchestration.WorkflowFailure",
      "subject": "/s1234567/datahub/runtime",
      "dataVersion": 1
    },
    "data": {
      "sourceId": "cf1e80db-3c1a-4b7b-88b7-2518b229603a",
      "tenant": "s1234567",
      "installation": "datahub",
      "environment": "runtime",
      "notificationId": "4fd01f17-4403-4a20-be86-3728611b4ad4",
      "notificationChannel": "0bf43704-8ac7-4ca8-86e1-155de3e56389",
      "notificationLevel": "WARN|ERROR",
      "notificationType": "OPERATIONAL",
      "notificationTime": "2022-01-13T12:50:05.358Z",
      "notificationSource": "AdeCore/WorkflowOrchestration",
      "contentText": "Workflow MY_DAG failed: Some problem description.",
      "contentType": "WorkflowFailure",
      "contentLink": "https://external.runtime.datahub.s1234567.agiledataengine.com/dagger/graph?dag_id=MY_DAG&execution_date=2022-01-12T10%3A16%3A58.119141%2B00%3A00"
    }
  }
]