Notify API
Notify API is used for queuing incoming source data files to be loaded into the target database in Agile Data Engine. Notify API has to be called from an external service, e.g. an integration tool, when source files are in cloud storage ready to be loaded into the data warehouse. See an example process and solutions below.
Additionally, a Swagger UI is provided which enables manual Notify API calls from a user interface. The user interface is useful for development and testing purposes, and contains the latest usage information.
See also:
Usage
API URLs are in the following format:
{baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/{sourceSystemName}/source-entities/{sourceEntityName}/manifests
where {baseUrl} is:
https://external-api.{environment}.datahub.{tenant}.saas.agiledataengine.com
{sourceSystemName} is the source system name defined in the SOURCE entity and {sourceEntityName} is the name of the SOURCE entity.
In the base URL:
{tenant} is the SaaS tenant, e.g. s1234567.
{environment} is the Runtime environment name, e.g. dev, test or prod.
Tenant name is part of the URL of your Agile Data Engine account. Environment names are listed in the Designer front page under Environments.
Authentication
All requests have to be authenticated. When calling the APIs from an external service, authentication is done with an API key and secret which are provided by the Agile Data Engine support team. Also, the public IP address of the service executing the requests has to be allowed by the support team.
Authentication in the user interface:
Navigate to the UI from the RUNTIMES menu. Select the desired Runtime environment and press Notify API.
Authorize the requests by pressing Authorize, then again Authorize under OAuth2.
Interfaces
Manifests
Operation | HTTP | URL | Description |
|---|---|---|---|
GET | {baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/{sourceSystemName}/source-entities/{sourceEntityName}/manifests | Search manifests for a source entity. Search can be filtered by defining manifest state. | |
GET | {baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/{sourceSystemName}/source-entities/{sourceEntityName}/manifests/{id} | Get information of a single manifest. | |
POST | {baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/{sourceSystemName}/source-entities/{sourceEntityName}/manifests | Create a new manifest. | |
POST | {baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/{sourceSystemName}/source-entities/{sourceEntityName}/manifests/{id}/notify | Notify ADE that manifest is ready to be processed. |
Manifest entries
Operation | HTTP | URL | Description |
|---|---|---|---|
GET | {baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/{sourceSystemName}/source-entities/{sourceEntityName}/manifests/{id}/entries | Get entry information of a manifest. | |
GET | {baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/{sourceSystemName}/source-entities/{sourceEntityName}/manifests/{id}/entries/{entryId} | Get information of a single manifest entry. | |
POST | {baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/{sourceSystemName}/source-entities/{sourceEntityName}/manifests/{id}/entries | Create a new entry into a manifest. | |
PUT | {baseUrl}/notify-api/tenants/local/installations/local/environments/local/source-systems/{sourceSystemName}/source-entities/{sourceEntityName}/manifests/{id}/entries | Create all entries into a manifest in one request. |
Responses
Code | Description |
|---|---|
200 | OK |
201 | Created |
401 | Unauthorized |
403 | Forbidden |
404 | Not Found |
504 | Gateway Timeout |
GET requests should get response code 200 when successful
POST and PUT requests should get response code 201 when successful
401 in the user interface usually indicates that the authorization has expired. Go back to Authorize, Logout and Authorize again.
504 is usually network related. Check that the API call is originating from an allowed IP address. You can request IP addresses to be allowed from the ADE Support Portal.
Schemas
Manifests
Property | Auto | Value | Default | Applicable format | Description |
|---|---|---|---|---|---|
batch | manifest batch id | all | Identifier of the data batch from the process that generated the file. Note that batch ids can be given either on the manifest level (all files are part of the same batch) or on the manifest entry level (files are separate batches). Batch ids are essential with the Run ID Logic. | ||
columns | array of column names | csv | SNOWFLAKE: List of column names in CSV files. Can be used when the CSV file has less columns than the target staging table or when the column order is different. | ||
compression | x* | BZIP2, GZIP, LZOP | all | Data file compression. Note that supported compression methods depend on the target database product. | |
created | x | manifest creation time | all | Automatically generated when manifest is created and should not be populated when creating manifest. | |
delim | COMMA, HASH, PIPE, SEMICOLON, TAB | COMMA | csv | Field delimiter of a CSV formatted data file. | |
format | x* | CSV, JSON, XML, UNKNOWN | all | Data file type. Use UKNOWN for other than listed file formats when notifying ADE and then OPT_FILE_FORMAT_OPTIONS to configure the correct file format in the file load. Note that supported formats depend on the target database product. | |
fullscanned | true/false | false | all | Describes if the manifest represents the full data from the source or just a part of it. When set to true, the target table is truncated before the file load. | |
id | x | manifest unique id | all | Automatically generated when manifest is created and should not be populated when creating manifest. | |
modified | x | manifest last modified time | all | Automatically updated when manifest changes and should not be populated when creating/updating manifest. | |
skiph | number of header lines to skip | 0 | csv | Number of header rows in a CSV formatted data file which should be skipped in the file load. | |
state | x | state of manifest | all | Only manifests in OPEN state can be modified through the API. OPEN: Manifest is created and is open to changes. I.e. you can still add more entries or change other properties of manifest. NOTIFIED: Manifest is sent to further processing and cannot be changed anymore. |
\*Recognized automatically from manifest entry file extensions if not given in the manifest.
Manifest entries
Property | Automatic | Value | Default | Description |
|---|---|---|---|---|
batch | file batch id | Identifier of the data batch from the process that generated the file. Note that batch ids can be given either on the manifest level (all files are part of the same batch) or on the manifest entry level (files are separate batches). Batch ids are essential with the Run ID Logic. | ||
id | x | unique manifest entry id | Automatically generated when a manifest entry is created and should not be populated when creating manifest entry. | |
sourceFile | entry filename and path | Entry (file) to be loaded. The path depends on where source entries are located (e.g. S3 or Azure Storage) and how the file load and the target database are configured. |
Notes
Inform the Agile Data Engine support team about the public IP addresses of the services that will be calling the Notify API. Requests are allowed from listed IPs only.
It is recommended to first test the notification process manually in the user interface. Once the process is tested and working, it is easier to adapt it to an external service that will run the API requests.
Examples
Notification process
Source integrations produce source data files into cloud storage.
File created events are captured OR file addesses are in some way transmitted to a notifier application.
The notifier application posts manifests into Agile Data Engine Notify API. The manifests contain file format settings and file addresses.
Agile Data Engine generates file load SQL statements and orchestrates them in the target database.
The target database executes file loads from cloud storage to staging tables.
CLI tool
A command line interface (CLI) tool for using Notify API is available in Github.
Python library
A Python library for implementing notifier solutions is maintained in Github.
Notifier examples
Example solutions for notifying Agile Data Engine are available in Github: