Skip to main content
Main navigation
Close navigation
Main
Main
What is Agile Data Engine
Key Concepts and Architecture
Getting Started
Tenant and Environments
Accessing ADE
Configuring ADE
Importing Initial Metadata
Setting Up Data Ingestion
Getting started with Notify API
Data Ingestion with External Tools
API Access and Setup
Configuration Guide
Admin UI
Configuration Packages
CONFIG_AUTOCOMPLETIONS
CONFIG_DATATYPES
INLINE DATATYPE
CONFIG_ENTITY_DEFAULTS
CONFIG_ENTITY_PERMISSIONS
CONFIG_ENVIRONMENT_VARIABLES
CONFIG_ISSUE_TRACK_SYSTEM
CONFIG_LOAD_SCHEDULES
CONFIG_LOAD_TEMPLATES
FTL Template Reference
Product-defined FTL templates
Supported FTL functions and macros
loads (provided FTL library)
sql (provided FTL library)
utils (provided FTL library)
CONFIG_LOAD_TRANSFORMATIONS
CONFIG_SYSTEMS
CONFIG_VALIDATIONS
Microsoft Entra ID Authentication Integration
Microsoft Entra ID Authentication Integration (User Management v1)
Private Connectivity and VPN
Role-Based Access Control (RBAC)
Target Database Setup
Amazon Redshift
Azure SQL Database
Azure Synapse SQL
Databricks SQL
Google BigQuery
Microsoft Fabric
Snowflake
User Guide
Data Warehouse Modeling
Data Vault
Hub
Link
Satellite
Status satellite
Multi-active satellite
Point-in-Time (PIT)
Transactional link
Dimensional
Fact
Dimension
SCD Type 2 with FTL Templates
Persistent staging
Customizing default attributes and transformations
Solution examples
Customizing File Loads in Snowflake
Dynamic tables in Snowflake
Hybrid table support in Snowflake
Load steps with user-defined functions
Sensitive Data Vault with FTL Templates
Staging with streaming tables in Databricks
Unloading data to cloud storage
Using Google Cloud services with API_CALL
Working with data shares
Designing Packages
Designing Workflows
From Metadata to DDL
Loading Source Data
Setting Up a File Notifier Application
Using Metadata Interface
Using Run ID Logic
FAQ
Reference Guide
Packages
Commit Package
Export Packages
Import Package
Package Actions
Package Change Comparison
Package Dependencies
Entities
Clone entity
Import entities
Move to another package
Entity types
Attributes
COLLATION VALUE
MASKING POLICY
TECHNICAL ATTRIBUTE TYPE
Database Tags
Configuration of Database Tags
Distribution styles
ALL
EVEN
KEY
Keys & References
DISTRIBUTION_KEY
FOREIGN_KEY
INTERLEAVED_SORT_KEY
PARTITION_KEY
PRIMARY_KEY
SEARCH_INDEX_KEY
SECONDARY_INDEX
SORT_KEY
UNIQUE_KEY
Permissions
Row-level permissions
Configuration of Row-level permissions
Physical types
DYNAMIC_TABLE
EXTERNAL_TABLE
ICEBERG_TABLE
MATERIALIZED_VIEW
METADATA_ONLY
SEMANTIC_VIEW
TABLE
VIEW
Physical options
DEPLOYMENT.COMPUTE_CLUSTER
REFRESH.DYNAMIC_TABLE_REFRESH_MODE
REFRESH.DYNAMIC_TABLE_TARGET_LAG
REFRESH.DYNAMIC_TABLE_WAREHOUSE
REFRESH.EXTERNAL_TABLE_AUTO_REFRESH
STORAGE.AUTO_CLUSTERING_SUSPEND
STORAGE.CHANGE_TRACKING
STORAGE.COLUMN_STORE
STORAGE.DYNAMIC_TABLE_BACKFILL_FROM
STORAGE.DYNAMIC_TABLE_IMMUTABLE_WHERE
STORAGE.DYNAMIC_TABLE_INITIALIZE_BEHAVIOR
STORAGE.DYNAMIC_TABLE_SIMPLE_ENTITY_RECREATE
STORAGE.EXTERNAL_TABLE_LOCATION
STORAGE.EXTERNAL_TABLE_FILE_FORMAT
STORAGE.EXTERNAL_TABLE_INTEGRATION
STORAGE.EXTERNAL_TABLE_FILENAME_PATTERN
STORAGE.EXTERNAL_TABLE_USER_SPEC_PARTITION
STORAGE.EXTERNAL_TABLE_REFRESH_ON_CREATE
STORAGE.EXTERNAL_TABLE_AWS_SNS_TOPIC
STORAGE.ICEBERG_EXTERNAL_LOCATION
STORAGE.ICEBERG_POINTER_FILE_STAGE
STORAGE.ICEBERG_SERIALIZATION_POLICY
STORAGE.PARTITION_SCHEME
STORAGE.REQUIRE_PARTITION_FILTER
STORAGE.SEARCH_OPTIMIZATION
STORAGE.SEARCH_OPTIMIZATION_OPTIONS
STORAGE.SECURE_VIEW
STORAGE.SORTKEY_RECLUSTER_IN_ENTITY_RECREATE
STORAGE.TABLE_STATISTICS_ATTRIBUTES
STORAGE.TABLE_STATISTICS_OPTIONS
STORAGE.TABLE_STATISTICS_UPDATE_IN_ENTITY_RECREATE
STORAGE.TABLE_STATISTICS_WHERE
STORAGE.TIME_TRAVEL_RETENTION_IN_DAYS
STORAGE.TRANSIENT_TABLE
Loads
Entity mapping
Entity mapping types
Import mappings
Load languages
SQL
API_CALL
Schema
Load options
CUSTOM
OPT_DATA_FILE_LOCATION
OPT_DELTA_EXTRACT
OPT_FILE_FORMAT_OPTIONS
OPT_HISTORIZED_BY_DATAHASH_AS_INSERT
OPT_INSERT_UPDATE_BY_DATAHASH_AS_MERGE
OPT_INSERT_UPDATE_BY_DATAHASH_AS_OVERWRITE
OPT_INSERT_UPDATE_BY_DATAHASH_AS_SWAP
OPT_INSERT_UPDATE_BY_ID_AS_MERGE
OPT_INSERT_UPDATE_BY_ID_AS_OVERWRITE
OPT_INSERT_UPDATE_BY_ID_AS_SWAP
OPT_INSERT_UPDATE_DELETE_BY_DATAHASH_AS_MERGE
OPT_INSERT_UPDATE_DELETE_BY_DATAHASH_AS_OVERWRITE
OPT_INSERT_UPDATE_DELETE_BY_DATAHASH_AS_SWAP
OPT_INSERT_UPDATE_DELETE_BY_ID_AS_MERGE
OPT_INSERT_UPDATE_DELETE_BY_ID_AS_OVERWRITE
OPT_INSERT_UPDATE_DELETE_BY_ID_AS_SWAP
OPT_LOAD_POOL
OPT_LOAD_RETRY
OPT_LOAD_WITH_BQ_API
OPT_LOCK_TIMEOUT_IN_SECS
OPT_LOCK_WAIT_TIME_IN_SECS
OPT_NOT_EXISTS_FROM
OPT_NOT_EXISTS_WHERE
OPT_NUM_RUN_IDS_PER_TRANSACTION
OPT_ORDER_BY
OPT_SCAN_ONLY_THE_LATEST_BY_KEY
OPT_SCAN_ONLY_THE_LATEST_WITH_SOURCE_ATTRIBUTE_TYPE
OPT_SF_FILE_COPY_OPTIONS
OPT_SKIP_DISTINCT_ONLY
OPT_SKIP_DUPLICATE_PREVENTION
OPT_STORAGE_INTEGRATION
OPT_USE_RUN_IDS
OPT_USE_RUN_IDS_INCREMENTALLY
OPT_USE_TIMESLOT_LOGIC
OPT_WHERE
Load step types
GATEKEEPER
OVERRIDE
OVERRIDE_FILE_LOAD
POST
PRE
SMOKE_BLACK
SMOKE_GREY
Load types
LOAD_FILE
TRANSFORM_PERSIST
TRANSFORM_SQL_VIEW
Variables
copy_options
dag_generation_mode
data_source
entity_lock_timeout_in_secs
entity_partition_key
entity_partition_key_{N}
env_short_name
environment_name
file_format
from_file
from_file_list
from_path
load_start_timemillis
loadablerunids
loadname
manifestrunid
multisource_mode
packagename
packageversion
scheduling_name
source_credentials
source_encryption
source_entity_attribute_by_{attribute_type}
source_entity_attribute_list
source_entity_name
source_entity_name_{N}
source_entity_schema
source_entity_schema_{N}
source_system_name
sourcebatchidattr (deprecated)
sourcerunidattr
stagingbatchidattr (deprecated)
storage_integration
system.deployment.entity_deletion.method
system.deployment.entity_deletion.method.{physical_type}
system.deployment.entity_recreate.dependent_dynamic_table.refresh
system.runtime.load_retry
target_db_name
target_entity_attribute_by_{attribute_type}
target_entity_attribute_list
target_entity_attribute_list_with_transform
target_entity_attribute_list_with_transform_cast_and_positions
target_entity_attribute_list_with_transform_cast_and_names
target_entity_attribute_list_with_transform_cast_exttable
target_entity_attribute_list_with_transform_cast_exttable_parquet
target_entity_attribute_list_with_transform_cast_parquet
target_entity_logical_name
target_entity_name
target_entity_transformation_by_{attribute_type}
target_schema
targetbatchid (deprecated)
targetbatchidattr (deprecated)
targetrunid
targetrunidattr
timemillis
warehouse_name
Deployment Management
Deployments UI
Designer main view
Deployments view
Package view
Workflow Orchestration
DAG generation mode
Pools
System DAGs
Triggering a Workflow
Workflow Preconditions
Insights
Access and Roles
DataOps Management KPIs
Insights Settings
Documentation Rules
Object Naming Rules
User and Tag Groups
Notify API
Manifests
Create manifest (POST)
Get manifest (GET)
Notify manifest (POST)
Search manifests (GET)
Manifest entries
Create entry (POST)
Create multiple entries (PUT)
Get entries (GET)
Get entry (GET)
External API
Notification Events
AWS SNS
Azure Event Grid
Google Pub/Sub
Notification Types v1
Release Notes
Agile Data Engine 26.0
Release Notes 26.0
Release Notes 26.0.645
Agile Data Engine 25.2
Release Notes 25.2
Release Notes 25.2.639
Release Notes 25.2.640
Release Notes 25.2.641
Release Notes 25.2.642
Release Notes 25.2.643
Agile Data Engine 25.1
Release Notes 25.1.632.patch13
Release Notes 25.1.September.2
Release Notes 25.1.632.patch12
Release Notes 25.1.September.1
Release Notes 25.1.632.patch11
Release Notes 25.1.632.patch10
Release Notes 25.1.632.patch9
Release Notes 25.1.July.2
Release Notes 25.1.July.1
Release Notes 25.1.632.patch8
Release Notes 25.1.632.patch6
Release Notes 25.1.June.1
Release Notes 25.1.May.1
Release Notes 25.1.632.patch5
Release Notes 25.1.632.patch4
Release Notes 25.1.632.patch3
Release Notes 25.1.632.patch2
Release Notes 25.1.632.patch1
Release Notes 25.1.632
Release Notes 25.1
Agile Data Engine 25.0
Release Notes 25.0.April.3
Release Notes 25.0.April.2
Release Notes 25.0.629
Release Notes 25.0.April.1
Release Notes 25.0.628
Release Notes 25.0.627.patch3
Release Notes 25.0.627.patch1
Release Notes 25.0
Agile Data Engine 24.0
Release Notes 24.0.569
Release Notes 24.0.560
Release Notes 24.0.553
Release Notes 24.0
Agile Data Engine 23.1
Release Notes 23.1.540
Release Notes 23.1.534
Release Notes 23.1.532
Release Notes 23.1
Agile Data Engine 23.0
Release Notes 23.0.MARCH for External API
Release Notes 23.0.492.patch1
Release Notes 23.0
Agile Data Engine 22.1
Release Notes 22.1.11.patch19
Release Notes 22.1.11.patch16
Release Notes 22.1.11.patch15
Release Notes 22.1.11.patch14
Release Notes 22.1.11.patch13
Release Notes 22.1.11.patch12
Release Notes 22.1.11.patch11
Release Notes 22.1.11.patch10
Release Notes 22.1.11.patch9
Release Notes 22.1.11.patch8
Release Notes 22.1.11.patch7
Release Notes 22.1.11.patch6
Release Notes 22.1.11.patch5
Release Notes 22.1.11.patch4
Release Notes 22.1.11.patch3
Release Notes 22.1.11.patch2
Release Notes 22.1.11.patch1
Release Notes 22.1.11
Release Notes 22.1.10
Release Notes 22.1
Agile Data Engine 22.0
Release Notes 22.0.469
Release Notes 22.0.468
Release Notes 22.0.467.patch1
Release Notes 22.0.467
Release Notes 22.0.466
Release Notes 22.0.464
Release Notes 22.0.460
Release Notes 22.0
Agile Data Engine 21.0
Release Notes 21.0.443.patch25
Release Notes 21.0.443.patch24
Release Notes 21.0.443.patch23
Release Notes 21.0.443.patch22
Release Notes 21.0.443.patch21
Release Notes 21.0.443.patch20
Release Notes 21.0.443.patch19
Release Notes 21.0.443.patch18
Release Notes 21.0.443.patch17
Release Notes 21.0.443.patch16
Release Notes 21.0.443.patch15
Release Notes 21.0.443.patch14
Release Notes 21.0.443.patch13
Release Notes 21.0.443.patch12
Release Notes 21.0.443.patch11
Release Notes 21.0.443.patch10
Release Notes 21.0.443.patch9
Release Notes 21.0.443.patch8
Release Notes 21.0.443.patch7
Release Notes 21.0.443.patch6
Release Notes 21.0.443.patch5
Release Notes 21.0.443.patch4
Release Notes 21.0.443.patch3
Release Notes 21.0.443.patch2
Release Notes 21.0.443.patch1
Release Notes 21.0.443
Release Notes 21.0.435.patch18
Release Notes 21.0.435.patch17
Release Notes 21.0.435.patch16
Release Notes 21.0.435.patch15
Release Notes 21.0.435.patch14
Release Notes 21.0.435.patch13
Release Notes 21.0.435.patch12
Release Notes 21.0.435.patch11
Release Notes 21.0.435.patch10
Release Notes 21.0.435.patch9
Release Notes 21.0.435.patch8
Release Notes 21.0.435.patch7
Release Notes 21.0.435.patch6
Release Notes 21.0.435.patch5
Release Notes 21.0.435.patch4
Release Notes 21.0.435.patch3
Release Notes 21.0.435.patch2
Release Notes 21.0.435.patch1
Release Notes 21.0.435
Release Notes 21.0.420.patch8
Release Notes 21.0.420.patch7
Release Notes 21.0.420.patch6
Release Notes 21.0.420.patch5
Release Notes 21.0.420.patch4
Release Notes 21.0.420.patch3
Release Notes 21.0.420.patch2
Release Notes 21.0.420.patch1
Release Notes 21.0.420
Release Notes 21.0
Agile Data Engine 20.2
Release Notes 20.2.396.patch11
Release Notes 20.2.396.patch10
Release Notes 20.2.396.patch9
Release Notes 20.2.2.396.patch8
Release Notes 20.2
Agile Data Engine 20.1
Release Notes 20.1
Agile Data Engine 20.0
Release Notes 20.0
Breadcrumbs
Reference Guide
Loads
On this Page
Load types
LOAD_FILE
TRANSFORM_PERSIST
TRANSFORM_SQL_VIEW
