ADE Notifier Concepts

This guide will explain the terminology and connection between Notify API and ADE Designer.

See also:

Terminology

Notify API
- Agile Data Engine’s API for notifying files.
Source entity
- Entity created in ADE with entity type SOURCE
Source system
- Source system create in CONFIG_SYSTEMS-package
Manifest
- Representation of single notifiable object
- Consists two parts:
  - Header: contains metadata about the files per source entity, such as delimiter, format and compression of files per source entity.
  - Entry: contains file paths in cloud storage. One manifest can contain 1-n entries.
Notify
- This means closing the manifest, i.e. telling ADE that the manifest and its entries are ready to be loaded to target database.
Notifier application
- Set of services in customer cloud, which will implement the logic to send file events to Notify API.

ADE Designer and Notify API relation

As specified in Notify API usage, Notify API uses composite key sourceSystemName + sourceEntityName in API calls. These attributes map into SOURCE-entities in ADE.

For example, the following URL will match to the following entity in ADE Designer.

sourceSystemName = digitraffic
sourceEntityName = metadata_vessels

Notify API URL:

source-systems/digitraffic/source-entities/metadata_vessels/manifests

In Designer, those are related to SOURCE SYSTEM and ENTITY NAME -metadata attributes in entity where entity type is SOURCE.

sourceSystemName → SOURCE SYSTEM
sourceEntityName → ENTITY NAME

What does Notifier actually do?

Manifests and entries that are added to Notify API, will be used by ADE to compile load command to target database. See examples below.

Snowflake

ADE Notify API was notified for source system digitraffic and entity locations_latest. The Notify API URL for that was:

source-systems/digitraffic/source-entities/locations_latest/manifests

Request body for that manifest was:

JSON

{
    "fullscanned": false,
    "delim": "SEMICOLON",
    "skiph": 1,
    "compression": "GZIP",
    "format": "CSV"
}

For this manifest, the following entries were added:

CODE

[
  {
    "sourceFile": "@staging.ade_prof_aws_stage/digitraffic/locations_latest/2023/08/29/locations_latest_20230829_010008.csv.gz"
  },
  {
    "sourceFile": "@staging.ade_prof_aws_stage/digitraffic/locations_latest/2023/08/29/locations_latest_20230829_011508.csv.gz"
  },
  {
    "sourceFile": "@staging.ade_prof_aws_stage/digitraffic/locations_latest/2023/08/29/locations_latest_20230829_013008.csv.gz"
  },
  {
    "sourceFile": "@staging.ade_prof_aws_stage/digitraffic/locations_latest/2023/08/29/locations_latest_20230829_014508.csv.gz"
  }
]

The source entity in ADE is the following:

And the target entity the following:

When ADE Workflow is executed, ADE will compile the following COPY-command to Snowflake:

CODE

COPY INTO staging.STG_DIGITRAFFIC_LOCATIONS_LATEST
FROM '@staging.ade_prof_aws_stage/digitraffic/locations_latest/2023/08/29/'
FILES=('locations_latest_20230829_010008.csv.gz', 
       'locations_latest_20230829_011508.csv.gz', 
       'locations_latest_20230829_013008.csv.gz', 
       'locations_latest_20230829_014508.csv.gz')
FILE_FORMAT = (
  type='csv' 
  skip_header=1 
  field_delimiter=';' 
  FIELD_OPTIONALLY_ENCLOSED_BY='"' 
  COMPRESSION='GZIP'
);

Google BigQuery

ADE Notify API was notified for source system digitraffic and entity metadata_vessels_bq. The Notify API URL for that was:

source-systems/digitraffic/source-entities/metadata_vessels_bq/manifests

Request body for that manifest was:

JSON

{
    "fullscanned": false,
    "compression": "GZIP",
    "format": "JSON"
}

For this manifest, the following entries were added:

CODE

[
  {
    "sourceFile": "gs://adedemo-sourcedata-dev/digitraffic/metadata_vessels/2023/09/13/metadata_vessels.batch.1694606403527.json.gz" 
  },
  {
    "sourceFile": "gs://adedemo-sourcedata-dev/digitraffic/metadata_vessels/2023/09/13/metadata_vessels.batch.1694598004770.json.gz"
  },
  {
    "sourceFile": "gs://adedemo-sourcedata-dev/digitraffic/metadata_vessels/2023/09/13/metadata_vessels.batch.1694599204923.json.gz"
  }
]

The source entity in ADE is the following:

And the target entity the following:

When ADE Workflow is executed, ADE will compile the following LOAD DATA INTO -command to BigQuery:

CODE

LOAD DATA INTO staging_bq.STG_DIGITRAFFIC_METADATA_VESSELS_JSON(
  stg_batch_id INT64, 
  stg_create_time TIMESTAMP, 
  stg_source_system STRING, 
  stg_source_entity STRING, 
  stg_file_name STRING, 
  payload JSON) 
FROM FILES(
  format='JSON',
  compression='GZIP', 
  uris=[
    'gs://adedemo-sourcedata-dev/digitraffic/metadata_vessels/2023/09/13/metadata_vessels.batch.1694606403527.json.gz', 
    'gs://adedemo-sourcedata-dev/digitraffic/metadata_vessels/2023/09/13/metadata_vessels.batch.1694598004770.json.gz',
    'gs://adedemo-sourcedata-dev/digitraffic/metadata_vessels/2023/09/13/metadata_vessels.batch.1694599204923.json.gz'
  ]);