Skip to main content
Skip table of contents

Notify API

Notify API is used for queuing incoming source data files to be loaded into the target database in Agile Data Engine. Notify API has to be called from an external service, e.g. an integration tool, when source files are in cloud storage ready to be loaded into the data warehouse. See an example process and solutions below.

Additionally, a Swagger UI is provided which enables manual Notify API calls from a user interface. The user interface is useful for development and testing purposes, and contains the latest usage information.


See also:


Usage

API URLs are in the following format:

{baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/{sourceSystemName}/source-entities/{sourceEntityName}/manifests

where {baseUrl} is:

https://external-api.{environment}.datahub.{tenant}.saas.agiledataengine.com

{sourceSystemName} is the source system name defined in the SOURCE entity and {sourceEntityName} is the name of the SOURCE entity.

In the base URL:

  • {tenant} is the SaaS tenant, e.g. s1234567.

  • {environment} is the Runtime environment name, e.g. dev, test or prod.

Tenant name is part of the URL of your Agile Data Engine account. Environment names are listed in the Designer front page under Environments.


Authentication

All requests have to be authenticated. When calling the APIs from an external service, authentication is done with an API key and secret which are provided by the Agile Data Engine support team. Also, the public IP address of the service executing the requests has to be allowed by the support team.

Authentication in the user interface:

  1. Navigate to the UI from the RUNTIMES menu. Select the desired Runtime environment and press Notify API.

  2. Authorize the requests by pressing Authorize, then again Authorize under OAuth2.


Interfaces

Manifests

Operation

HTTP

URL

Description

Search manifests

GET

{baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/{sourceSystemName}/source-entities/{sourceEntityName}/manifests

Search manifests for a source entity. Search can be filtered by defining manifest state.

Get manifest

GET

{baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/{sourceSystemName}/source-entities/{sourceEntityName}/manifests/{id}

Get information of a single manifest.

Create manifest

POST

{baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/{sourceSystemName}/source-entities/{sourceEntityName}/manifests

Create a new manifest.

Notify manifest

POST

{baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/{sourceSystemName}/source-entities/{sourceEntityName}/manifests/{id}/notify

Notify ADE that manifest is ready to be processed.


Manifest entries

Operation

HTTP

URL

Description

Get entries

GET

{baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/{sourceSystemName}/source-entities/{sourceEntityName}/manifests/{id}/entries

Get entry information of a manifest.

Get entry

GET

{baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/{sourceSystemName}/source-entities/{sourceEntityName}/manifests/{id}/entries/{entryId}

Get information of a single manifest entry.

Create entry

POST

{baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/{sourceSystemName}/source-entities/{sourceEntityName}/manifests/{id}/entries

Create a new entry into a manifest.

Create multiple entries

PUT

{baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/{sourceSystemName}/source-entities/{sourceEntityName}/manifests/{id}/entries

Create all entries into a manifest in one request.


Responses

Code

Description

200

OK

201

Created

401

Unauthorized

403

Forbidden

404

Not found

  • GET requests should get response code 200 when successful

  • POST and PUT requests should get response code 201 when successful

  • 401 in the user interface usually indicates that the authorization has expired. Go back to Authorize, Logout and Authorize again.


Schemas

Manifests

Property

Auto

Value

Default

Applicable format

Description

batch

manifest batch id

all

Identifier of the data batch from the process that generated the file. Note that batch ids can be given either on the manifest level (all files are part of the same batch) or on the manifest entry level (files are separate batches). Batch ids are essential with the Run ID Logic.

columns

array of column names

csv

SNOWFLAKE: List of column names in CSV files. Can be used when the CSV file has less columns than the target staging table or when the column order is different.

compression

x*

BZIP2, GZIP, LZOP

all

Data file compression. Note that supported compression methods depend on the target database product.

created

x

manifest creation time

all

Automatically generated when manifest is created and should not be populated when creating manifest.

delim

COMMA, HASH, PIPE, SEMICOLON, TAB

COMMA

csv

Field delimiter of a CSV formatted data file.

format

x*

CSV, JSON, XML, UNKNOWN

all

Data file type. Use UKNOWN for other than listed file formats when notifying ADE and then OPT_FILE_FORMAT_OPTIONS to configure the correct file format in the file load.

Note that supported formats depend on the target database product.

fullscanned

true/false

false

all

Describes if the manifest represents the full data from the source or just a part of it. When set to true, the target table is truncated before the file load.

id

x

manifest unique id

all

Automatically generated when manifest is created and should not be populated when creating manifest.

modified

x

manifest last modified time

all

Automatically updated when manifest changes and should not be populated when creating/updating manifest.

skiph

number of header lines to skip

0

csv

Number of header rows in a CSV formatted data file which should be skipped in the file load.

state

x

state of manifest

all

Only manifests in OPEN state can be modified through the API.

OPEN: Manifest is created and is open to changes. I.e. you can still add more entries or change other properties of manifest.

NOTIFIED: Manifest is sent to further processing and cannot be changed anymore.

\*Recognized automatically from manifest entry file extensions if not given in the manifest.


Manifest entries

Property

Automatic

Value

Default

Description

batch

file batch id

Identifier of the data batch from the process that generated the file. Note that batch ids can be given either on the manifest level (all files are part of the same batch) or on the manifest entry level (files are separate batches). Batch ids are essential with the Run ID Logic.

id

x

unique manifest entry id

Automatically generated when a manifest entry is created and should not be populated when creating manifest entry.

sourceFile

entry filename and path

Entry (file) to be loaded. The path depends on where source entries are located (e.g. S3 or Azure Storage) and how the file load and the target database are configured.


Notes

Inform the Agile Data Engine support team about the public IP addresses of the services that will be calling the Notify API. Requests are allowed from listed IPs only.

It is recommended to first test the notification process manually in the user interface. Once the process is tested and working, it is easier to adapt it to an external service that will run the API requests.


Examples

Notification process

  • Source integrations produce source data files into cloud storage.

  • File created events are captured OR file addesses are in some way transmitted to a notifier application.

  • The notifier application posts manifests into Agile Data Engine Notify API. The manifests contain file format settings and file addresses.

  • Agile Data Engine generates file load SQL statements and orchestrates them in the target database.

  • The target database executes file loads from cloud storage to staging tables.

CLI tool

A command line interface (CLI) tool for using Notify API is available in Github.

Python library

A Python library for implementing notifier solutions is maintained in Github.

Notifier examples

Example solutions for notifying Agile Data Engine are available in Github:

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.