Setting Up Data Ingestion

With Agile Data Engine, you have different architectural options to ingest data into target database.

1. Ingesting data from cloud storage to target database with Agile Data Engine

2. Use data ingestion tool to load data into target database

Ingesting data from cloud storage with ADE

To ingest data from cloud storage to target database with ADE, there are pre-requisites to get that working. These are:

• Setting up ADE Notifier to customer cloud

  • This application will send information about file events to Notify API

  • Depending on the cloud, different cloud infrastructure components

When using Notify API, there are benefits, such as:

• ADE handles the whole pipeline from cloud storage to target database

• Logging the data ingestion process is in the same tool as data transformation

• Ability to use runid logic to boost the performance of delta data handling

In this case, ADE will handle data ingestion from cloud storage to target database. In addition, it will do other functionalities as normally, which are transformations, schema changes and workflow orchestration.

Using data ingestion tool to load data into target database

It is also possible to ingest data by using a dedicated data ingestion tool. This has some benefits, such as:

• You are not tied to ADE with your data ingestion process

• You can pick and choose any suitable tool

But also some drawbacks, such as:

• ADE runid logic can be used, but needs External API usage

• You have to monitor and orchestrate data ingestion in a separate tool

• Synchronization of concurrent deployments and loads is lost, since data ingestion and other data loads are in separate tools

In this case, ADE will handle transformations, schema changes and workflow orchestration, but the data ingestion is out of scope.