Skip to main content
Skip table of contents

Getting started with Notify API

Notify API is a key component in how ADE manages file-based data ingestion from cloud storage into the target database. It allows external systems to notify ADE when new data files are ready to be processed. Each notification includes file format metadata and file paths, packaged into a structured message called a manifest, which informs ADE how to generate a file load command to the target database.

This guide introduces the core concepts of the notification process, provides links to related resources for deeper understanding, and includes examples that demonstrate how the file load process works in practice.


See also:


Terminology

Term

Description

Notify API

ADE API used to notify ADE about new files available for loading from cloud storage.

Source entity

An entity created in ADE with the entity type set to SOURCE. Represents a logical source table or file structure.

Source system

A source system defined in the CONFIG_SYSTEMS configuration package. Used to group and manage related source entities.

Manifest

A manifest is a notification message sent to the Notify API, containing metadata and file paths related to loadable source data files. It groups one or more files (entries) under a shared context (body) that defines how the files should be interpreted and processed by ADE.

A single manifest can only contain entries for one source entity.

  • Body: Metadata about the files, such as format, delimiter, and compression.

  • Entries: One or more file paths pointing to data files in cloud storage.

Notify

Closing a manifest, i.e. informing ADE that all file entries are complete and the manifest is ready to be processed and loaded.

Notifier application

A set of services deployed in the customer's cloud environment that detects file events and posts them to the Notify API.


How Notify API maps to SOURCE entities

In Notify API calls, the combination of source system name and source entity name in the request URL identify the SOURCE entity.

For example, the following Notify API URL:

{baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/source-systems/digitraffic/source-entities/metadata_vessels/manifests

Maps to an entity with:

  • Entity type = SOURCE

  • Source system = DIGITRAFFIC

  • Entity name = METADATA_VESSELS

Screenshot 2025-07-22 at 10.34.04.png

Note that the Notify API URLs are always in lower case, see more details in the Notify API reference.


Examples

Manifests and entries posted to Notify API will be used by ADE to generate load commands to the target database.

See examples:

Snowflake

ADE Notify API is called to create a manifest for source system digitraffic and entity locations_latest.

Notify API URL:

{baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/source-systems/digitraffic/source-entities/locations_latest/manifests

Manifest body:

JSON
{
    "fullscanned": false,
    "delim": "SEMICOLON",
    "skiph": 1,
    "compression": "GZIP",
    "format": "CSV"
}

Entries added to the manifest:

JSON
[
  {
    "sourceFile": "@staging.ade_prof_aws_stage/digitraffic/locations_latest/2023/08/29/locations_latest_20230829_010008.csv.gz"
  },
  {
    "sourceFile": "@staging.ade_prof_aws_stage/digitraffic/locations_latest/2023/08/29/locations_latest_20230829_011508.csv.gz"
  },
  {
    "sourceFile": "@staging.ade_prof_aws_stage/digitraffic/locations_latest/2023/08/29/locations_latest_20230829_013008.csv.gz"
  },
  {
    "sourceFile": "@staging.ade_prof_aws_stage/digitraffic/locations_latest/2023/08/29/locations_latest_20230829_014508.csv.gz"
  }
]

SOURCE entity in the Designer:

Screenshot 2025-07-22 at 13.22.01.png

STAGE table file load in the Designer, with the SOURCE entity mapped:

image-20250722-102808.png

When the load is executed, ADE runs the following COPY INTO command in Snowflake:

SQL
COPY INTO staging.STG_DIGITRAFFIC_LOCATIONS_LATEST
FROM '@staging.ade_prof_aws_stage/digitraffic/locations_latest/2023/08/29/'
FILES=('locations_latest_20230829_010008.csv.gz', 
       'locations_latest_20230829_011508.csv.gz', 
       'locations_latest_20230829_013008.csv.gz', 
       'locations_latest_20230829_014508.csv.gz')
FILE_FORMAT = (
  type='csv' 
  skip_header=1 
  field_delimiter=';' 
  FIELD_OPTIONALLY_ENCLOSED_BY='"' 
  COMPRESSION='GZIP'
);
Google BigQuery

ADE Notify API is called to create a manifest for source system digitraffic and entity metadata_vessels_bq.

Notify API URL:

{baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/source-systems/digitraffic/source-entities/metadata_vessels_bq/manifests

Manifest body:

JSON
{
    "fullscanned": false,
    "compression": "GZIP",
    "format": "JSON"
}

Entries added to the manifest:

JSON
[
  {
    "sourceFile": "gs://adedemo-sourcedata-dev/digitraffic/metadata_vessels/2023/09/13/metadata_vessels.batch.1694606403527.json.gz" 
  },
  {
    "sourceFile": "gs://adedemo-sourcedata-dev/digitraffic/metadata_vessels/2023/09/13/metadata_vessels.batch.1694598004770.json.gz"
  },
  {
    "sourceFile": "gs://adedemo-sourcedata-dev/digitraffic/metadata_vessels/2023/09/13/metadata_vessels.batch.1694599204923.json.gz"
  }
]

SOURCE entity in the Designer:

image-20250722-103426.png

STAGE table file load in the Designer, with the SOURCE entity mapped:

image-20250722-103501.png

When the load is executed, ADE runs the following LOAD DATA INTO statement in BigQuery:

SQL
LOAD DATA INTO staging_bq.STG_DIGITRAFFIC_METADATA_VESSELS_JSON(
  stg_batch_id INT64, 
  stg_create_time TIMESTAMP, 
  stg_source_system STRING, 
  stg_source_entity STRING, 
  stg_file_name STRING, 
  payload JSON) 
FROM FILES(
  format='JSON',
  compression='GZIP', 
  uris=[
    'gs://adedemo-sourcedata-dev/digitraffic/metadata_vessels/2023/09/13/metadata_vessels.batch.1694606403527.json.gz', 
    'gs://adedemo-sourcedata-dev/digitraffic/metadata_vessels/2023/09/13/metadata_vessels.batch.1694598004770.json.gz',
    'gs://adedemo-sourcedata-dev/digitraffic/metadata_vessels/2023/09/13/metadata_vessels.batch.1694599204923.json.gz'
  ]);
JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.