Variables
Agile Data Engine supports predefined and user-defined variables to enable customization in various different use cases.
See also:
Usage
Variables are referred to by encapsulating the variable name in angle brackets, for example:
<variable_name>Notes
If environment variables are used in loads and changes are made to the environment variables, the loads must also be redeployed for the changes to take effect.
Variable types
Level | Type | Example | Scope |
|---|---|---|---|
0 | Predefined environment level variables | Environment | |
1 | Environment variables in ADE main configuration (Private Edition) |
| Environment |
2 | Environment variables in CONFIG_ENVIRONMENT_VARIABLES |
| Environment |
3 | Predefined schedule level variables | Workflow | |
4 | Schedule level variables in a configuration package |
| Workflow |
5 | Predefined load level variables | Load |
Predefined variables
See list of predefined variables below. Depending on the variable, it can be used in:
Deployment: Control parts in deployment procedures such as grants.
Loads: Use variables in load step definitions.
Variable | Deployment | Loads | Description |
|---|---|---|---|
x | Resolved as the Snowflake copy options in file load execution. | ||
x | x | Resolved as the DAG generation mode name in load execution. | |
x | Synonymous to storage_integration, resolved as the storage integration (Snowflake) or external data source (Azure SQL Database) in file load execution. | ||
DISABLE_SRC_SYS_NAME_IN_FILE_NAME | x | x | Used for backward compatibility only. |
x | x | Deprecated. Alias to environment_name. Use environment_name instead. | |
x | x | Resolved as the environment name in deployments and in load execution. | |
x | Resolved as the file format options in file load execution. | ||
x | Resolved as the source file path in file load execution. | ||
x | Resolved as a comma-separated source file list in file load execution. | ||
x | Resolved as the source path(s) in file load execution. | ||
x | Resolved as the load start Unix timestamp in milliseconds in load execution. | ||
x | Resolved as a comma-separated list of loadable batch ids in load execution. Used with the Run ID Logic. | ||
| x | Resolved as the load name in load execution. | |
x | Populated with Run ID from manifest level in file load execution time. | ||
| x | Resolved as the package name in load execution. | |
| x | Resolved as the package version in load execution. | |
x | x | Resolved as the schedule name in load execution. | |
x | Resolved as data source credentials in file load execution. | ||
x | Resolved as encryption settings in file load execution. | ||
x | Resolves as comma-separated list of first mapped source entity attributes | ||
x | Resolved as the source entity name in load execution. | ||
x | Resolved as the source entity schema in load execution. | ||
x | Resolves as the source system name in load execution. Source system is carried only with SOURCE entity types and in case both mapped entities are of type SOURCE, target entity source system will be used | ||
| x | Deprecated. Resolved as the data source batch id attribute name in load execution. Used with the Run ID Logic. | |
| x | Resolved as the data source run id attribute name in load execution. Used with the Run ID Logic. | |
x | Deprecated. Resolved as the staging table batch id attribute name in load execution. Used with the Run ID Logic. | ||
x | Resolved as the storage integration (Snowflake) or as the external data source (Azure SQL Database). | ||
| x | Resolved as the target database name in load execution. | |
x | Resolved as a comma-separated list of target entity attributes in file load execution. | ||
x | Resolves as comma-separated list of attribute mappings with their transformations | ||
x | Resolves as comma-separated list of attribute mappings with their transformations and 1-1 mappings without transformation will be cast to the target attribute physical datatype with the target attribute name | ||
target_entity_attribute_list_with_transform_cast_and_positions | x | Resolves as comma-separated list of attribute mappings with their transformations. For Azure databases resolves only 1-1 mappings, not supported in Google BigQuery, Amazon Redshift and PostgreSQL. | |
x | Resolves as full target entity attribute list having the transformations based on the first SOURCE entity mapping. One-to-one mappings without transformation formula will be translated with table creation column definition syntax. | ||
target_entity_attribute_list_with_transform_cast_exttable_parquet | x | Resolves as full target entity attribute list having the transformations based on the first SOURCE entity mapping. One-to-one mappings without transformation formula will be translated as single-level parquet column references with datatype for external table creation context. | |
x | Resolves as full target entity attribute list having the transformations based on the first SOURCE entity mapping. One-to-one mappings without transformation formula will be translated as single-level parquet column references with datatype in select context. | ||
x | Resolved as the target entity logical name in load execution. | ||
x | Resolved as the target entity name in load execution. | ||
x | Resolved as the schema of the target entity in load execution. | ||
| x | Deprecated. Resolved as a new target batch id value in load execution. Used with the Run ID Logic. | |
| x | Deprecated. Resolved as the target table batch id attribute name in load execution. Used with the Run ID Logic. | |
| x | Resolved as a new target run id value in load execution. Used with the Run ID Logic. | |
| x | Resolved as the target table run id attribute name in load execution. Used with the Run ID Logic. | |
x | Resolved as the current Unix timestamp in milliseconds in load execution. |