With Agile Data Engine, you have different architectural options to ingest data into target database.
Ingesting data from cloud storage to target database with Agile Data Engine
Use data ingestion tool to load data into target database
Ingesting data from cloud storage with ADE
To ingest data from cloud storage to target database with ADE, there are pre-requisites to get that working. These are:
Setting up ADE Notifier to customer cloud
This application will send information about file events to Notify API
Depending on the cloud, different cloud infrastructure components
When using Notify API, there are benefits, such as:
ADE handles the whole pipeline from cloud storage to target database
Logging the data ingestion process is in the same tool as data transformation
Ability to use runid logic to boost the performance of delta data handling
In this case, ADE will handle data ingestion from cloud storage to target database. In addition, it will do other functionalities as normally, which are transformations, schema changes and workflow orchestration.
Using data ingestion tool to load data into target database
It is also possible to ingest data by using a dedicated data ingestion tool. This has some benefits, such as:
You are not tied to ADE with your data ingestion process
You can pick and choose any suitable tool
But also some drawbacks, such as:
You have to monitor and orchestrate data ingestion in a separate tool
Synchronization of concurrent deployments and loads is lost, since data ingestion and other data loads are in separate tools
In this case, ADE will handle transformations, schema changes and workflow orchestration, but the data ingestion is out of scope.