Data Ingestion with External Tools
Data ingestion to target database can be done with any suitable tool. In those cases, ADE can be used to connect to the existing data and handle the data development and operations from there.
When ADE is not used for data ingestion, these things need to be considered:
Monitoring the data ingestion process with suitable tool
Managing the schema of the landing tables
Managing the metadata definitions of landing tables in ADE
Adding existing tables to ADE
When raw data is already ingested to target database, you need to create entity metadata for this data. To do that, check importing initial metadata from existing data in target database.
Metadata definition of existing data
In this example, there is existing table raw.SRC_ORDERS in the target database. Metadata definition of the entity was imported with the entity import. The entity type of this is SOURCE and physical type as METADATA_ONLY, meaning entity is only metadata definition of the table.
Attributes of this table:
When the metadata definition exists in ADE, it is possible to use this entity in the following entities and loads.
Creating staging table for the raw data
In this example case, a staging-table is created for the raw data and additional staging attributes are added. Depending on the use case, this entity could also be done as a view, if the raw data is already persisted in the database.
Entity definition for this table:
Entity contains automatically added default staging attributes and mapped attributes from the source-table:
The generated SQL for the stating table looks like the following:
For the default staging attributes, transformations were added in the entity mapping. For example, for stg_source_entity a variable was used to describe source entity.