Key Concepts and Architecture
Platform modules
Main platform modules of Agile Data Engine are Designer, Deployment Management and Runtime.
Designer
Agile Data Engine Designer is Web based graphical UI used for designing the data models and data load definitions like transformations, business rules and dependencies for the data platform.
Designer enables iterative data and solution modeling, continuous deployment and facilitates the standardization of modeling and load deployments.
Designer has a web based graphical user interface used through an internet browser without any additional installations to the user's computers. Designer is used by the users for designing the data warehouse data model and load definitions. It is also the interface for the users to view the graphical representations of the designed models and data lineages.
Deployment Management
Deployment Management module manages the continuous deployment pipeline functionality for data warehouse solution content. It enables continuous and automated deployment of database entities (tables, views via SQL DDL), data load code (SQL DML) and workflows (DAG:s) into different runtime environments.
Deployment Management stores all metadata changes committed by the user into a central version control repository and manages the deployment process.
Module handles the SQL generation of data model and loads created in Designer. It supports multiple cloud databases and their SQL dialects.
Runtime
Runtime is a module used for operating the data warehouse data loads and deploying changes. Separate Runtime required for each warehouse environment used in data warehouse development (development, testing and production).
Workflow orchestration is the main component of Runtime. It is used for orchestrating and operating (e.g. running) the data loads.
Runtime is also used as a workflow monitoring and troubleshooting the data pipelines managed by Agile Data Engine.
Key metadata concepts
Package
A package in Agile Data Engine enables the modularisation of the data warehouse content (data models and loads) into manageable and deployable pieces. This helps to achieve agility with truly incremental and iterative development.
A package is a single module in ADE which includes different number of entities and data loads. All the changes made within a package are written into ADE’s metadata database.
Package is also a single unit of commit and deployment.
Entity
Entity is a core metadata object in Agile Data Engine. An entity refers to data warehouse tables, views or informational metadata objects (sources).
Building a data warehouse in Agile Data Engine is based on entities:
A data model is created by creating entities.
A data pipeline is created between the source entities and the target entities.
Each entity has metadata that provides information about the entity itself. The metadata of an entity can include:
Name, attributes, keys, physical options, load definition, test definition, and permissions
Load
Loads define how data is loaded to an entity. One entity can have multiple loads, i.e. you can load same entity from multiple sources. Loads are executed in the Runtime environment using Workflow orchestration.
Loads include:
Data mappings. Data mapping connects source attribute(s) to target attribute(s). If data conversions or data manipulations are needed, those are done within data mapping.
Load schedules. A load inherits the schedule of it’s source entity’s load(s) or the schedule can be overwritten.
Possible custom load steps. Gatekeeper-, pre-, post-, smoke_grey-, smoke_black- or override step
Workflow
The workflows are formed by Agile Data Engine according to schedules defined and the dependencies between different loads (entity mappings). The workflows consist of loads which are chained according a source and target entities. The loads having the same schedule are included into the same workflow.