Getting started with Notify API

Notify API is a key component in how ADE manages file-based data ingestion from cloud storage into the target database. It allows external systems to notify ADE when new data files are ready to be processed. Each notification includes file format metadata and file paths, packaged into a structured message called a manifest, which informs ADE how to generate a file load command to the target database.

This guide introduces the core concepts of the notification process, provides links to related resources for deeper understanding, and includes examples that demonstrate how the file load process works in practice.

See also:

Notify API - Notify API reference documentation
API Access and Setup - Getting access to Notify API
Setting Up a File Notifier Application - How to set up an application for the notifying process
Loading Source Data - Guide to file loads
ADE File Notifier Reference (GitHub) - Notifier solution example

Terminology

Term	Description
Notify API	ADE API used to notify ADE about new files available for loading from cloud storage.
Source entity	An entity created in ADE with the entity type set to `SOURCE`. Represents a logical source table or file structure.
Source system	A source system defined in the CONFIG_SYSTEMS configuration package. Used to group and manage related source entities.
Manifest	A manifest is a notification message sent to the Notify API, containing metadata and file paths related to loadable source data files. It groups one or more files (entries) under a shared context (body) that defines how the files should be interpreted and processed by ADE. A single manifest can only contain entries for one source entity. Body: Metadata about the files, such as format, delimiter, and compression. Entries: One or more file paths pointing to data files in cloud storage.
Notify	Closing a manifest, i.e. informing ADE that all file entries are complete and the manifest is ready to be processed and loaded.
Notifier application	A set of services deployed in the customer's cloud environment that detects file events and posts them to the Notify API.

How Notify API maps to SOURCE entities

In Notify API calls, the combination of source system name and source entity name in the request URL identify the SOURCE entity.

For example, the following Notify API URL:

{baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/source-systems/digitraffic/source-entities/metadata_vessels/manifests

Maps to an entity with:

Entity type = SOURCE
Source system = DIGITRAFFIC
Entity name = METADATA_VESSELS

Note that the Notify API URLs are always in lower case, see more details in the Notify API reference.

Examples

Manifests and entries posted to Notify API will be used by ADE to generate load commands to the target database.

See examples:

Snowflake

ADE Notify API is called to create a manifest for source system digitraffic and entity locations_latest.

Notify API URL:

{baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/source-systems/digitraffic/source-entities/locations_latest/manifests

Manifest body:

JSON

{
    "fullscanned": false,
    "delim": "SEMICOLON",
    "skiph": 1,
    "compression": "GZIP",
    "format": "CSV"
}

Entries added to the manifest:

JSON

[
  {
    "sourceFile": "@staging.ade_prof_aws_stage/digitraffic/locations_latest/2023/08/29/locations_latest_20230829_010008.csv.gz"
  },
  {
    "sourceFile": "@staging.ade_prof_aws_stage/digitraffic/locations_latest/2023/08/29/locations_latest_20230829_011508.csv.gz"
  },
  {
    "sourceFile": "@staging.ade_prof_aws_stage/digitraffic/locations_latest/2023/08/29/locations_latest_20230829_013008.csv.gz"
  },
  {
    "sourceFile": "@staging.ade_prof_aws_stage/digitraffic/locations_latest/2023/08/29/locations_latest_20230829_014508.csv.gz"
  }
]

SOURCE entity in the Designer:

STAGE table file load in the Designer, with the SOURCE entity mapped:

When the load is executed, ADE runs the following COPY INTO command in Snowflake:

SQL

COPY INTO staging.STG_DIGITRAFFIC_LOCATIONS_LATEST
FROM '@staging.ade_prof_aws_stage/digitraffic/locations_latest/2023/08/29/'
FILES=('locations_latest_20230829_010008.csv.gz', 
       'locations_latest_20230829_011508.csv.gz', 
       'locations_latest_20230829_013008.csv.gz', 
       'locations_latest_20230829_014508.csv.gz')
FILE_FORMAT = (
  type='csv' 
  skip_header=1 
  field_delimiter=';' 
  FIELD_OPTIONALLY_ENCLOSED_BY='"' 
  COMPRESSION='GZIP'
);

Google BigQuery

ADE Notify API is called to create a manifest for source system digitraffic and entity metadata_vessels_bq.

Notify API URL:

{baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/source-systems/digitraffic/source-entities/metadata_vessels_bq/manifests

Manifest body:

JSON

{
    "fullscanned": false,
    "compression": "GZIP",
    "format": "JSON"
}

Entries added to the manifest:

JSON

[
  {
    "sourceFile": "gs://adedemo-sourcedata-dev/digitraffic/metadata_vessels/2023/09/13/metadata_vessels.batch.1694606403527.json.gz" 
  },
  {
    "sourceFile": "gs://adedemo-sourcedata-dev/digitraffic/metadata_vessels/2023/09/13/metadata_vessels.batch.1694598004770.json.gz"
  },
  {
    "sourceFile": "gs://adedemo-sourcedata-dev/digitraffic/metadata_vessels/2023/09/13/metadata_vessels.batch.1694599204923.json.gz"
  }
]

SOURCE entity in the Designer:

STAGE table file load in the Designer, with the SOURCE entity mapped:

When the load is executed, ADE runs the following LOAD DATA INTO statement in BigQuery:

SQL

LOAD DATA INTO staging_bq.STG_DIGITRAFFIC_METADATA_VESSELS_JSON(
  stg_batch_id INT64, 
  stg_create_time TIMESTAMP, 
  stg_source_system STRING, 
  stg_source_entity STRING, 
  stg_file_name STRING, 
  payload JSON) 
FROM FILES(
  format='JSON',
  compression='GZIP', 
  uris=[
    'gs://adedemo-sourcedata-dev/digitraffic/metadata_vessels/2023/09/13/metadata_vessels.batch.1694606403527.json.gz', 
    'gs://adedemo-sourcedata-dev/digitraffic/metadata_vessels/2023/09/13/metadata_vessels.batch.1694598004770.json.gz',
    'gs://adedemo-sourcedata-dev/digitraffic/metadata_vessels/2023/09/13/metadata_vessels.batch.1694599204923.json.gz'
  ]);