Skip to main content
Main
    What is Agile Data Engine
    • Key Concepts and Architecture
    Getting Started
    • Tenant and Environments
    • Accessing ADE
    • Configuring ADE
    • Importing Initial Metadata
      • Setting Up Data Ingestion
      • Getting started with Notify API
      • Data Ingestion with External Tools
    • API Access and Setup
    Configuration Guide
    • Admin UI
      • Configuration Packages
      • CONFIG_AUTOCOMPLETIONS
        • CONFIG_DATATYPES
        • INLINE DATATYPE
      • CONFIG_ENTITY_DEFAULTS
      • CONFIG_ENTITY_PERMISSIONS
      • CONFIG_ENVIRONMENT_VARIABLES
      • CONFIG_ISSUE_TRACK_SYSTEM
      • CONFIG_LOAD_SCHEDULES
        • CONFIG_LOAD_TEMPLATES
          FTL Template Reference
          • Product-defined FTL templates
            • Supported FTL functions and macros
            • loads (provided FTL library)
            • sql (provided FTL library)
            • utils (provided FTL library)
      • CONFIG_LOAD_TRANSFORMATIONS
      • CONFIG_SYSTEMS
      • CONFIG_VALIDATIONS
      Microsoft Entra ID Authentication Integration
      • Microsoft Entra ID Authentication Integration (User Management v1)
    • Private Connectivity and VPN
    • Role-Based Access Control (RBAC)
      • Target Database Setup
      • Amazon Redshift
      • Azure SQL Database
      • Azure Synapse SQL
      • Databricks SQL
      • Google BigQuery
      • Microsoft Fabric
      • Snowflake
    User Guide
      Data Warehouse Modeling
        Data Vault
        • Hub
        • Link
        • Satellite
        • Status satellite
        • Multi-active satellite
        • Point-in-Time (PIT)
        • Transactional link
        Dimensional
        • Fact
          • Dimension
          • SCD Type 2 with FTL Templates
      • Persistent staging
      • Customizing default attributes and transformations
      Solution examples
      • Customizing File Loads in Snowflake
      • Dynamic tables in Snowflake
      • Hybrid table support in Snowflake
      • Load steps with user-defined functions
      • Sensitive Data Vault with FTL Templates
      • Staging with streaming tables in Databricks
      • Unloading data to cloud storage
      • Using Google Cloud services with API_CALL
      • Working with data shares
    • Designing Packages
    • Designing Workflows
    • From Metadata to DDL
    • Loading Source Data
    • Setting Up a File Notifier Application
    • Using Metadata Interface
    • Using Run ID Logic
    • FAQ
    Reference Guide
      Packages
      • Commit Package
      • Export Packages
      • Import Package
      • Package Actions
      • Package Change Comparison
      • Package Dependencies
      Entities
      • Clone entity
      • Import entities
      • Move to another package
      • Entity types
        • Attributes
        • COLLATION VALUE
        • MASKING POLICY
        • TECHNICAL ATTRIBUTE TYPE
        Database Tags
        • Configuration of Database Tags
        Distribution styles
        • ALL
        • EVEN
        • KEY
        Keys & References
        • DISTRIBUTION_KEY
        • FOREIGN_KEY
        • INTERLEAVED_SORT_KEY
        • PARTITION_KEY
        • PRIMARY_KEY
        • SEARCH_INDEX_KEY
        • SECONDARY_INDEX
        • SORT_KEY
        • UNIQUE_KEY
        Permissions
          Row-level permissions
          • Configuration of Row-level permissions
        Physical types
        • DYNAMIC_TABLE
        • EXTERNAL_TABLE
        • ICEBERG_TABLE
        • MATERIALIZED_VIEW
        • METADATA_ONLY
        • SEMANTIC_VIEW
        • TABLE
        • VIEW
        Physical options
        • DEPLOYMENT.COMPUTE_CLUSTER
        • REFRESH.DYNAMIC_TABLE_REFRESH_MODE
        • REFRESH.DYNAMIC_TABLE_TARGET_LAG
        • REFRESH.DYNAMIC_TABLE_WAREHOUSE
        • REFRESH.EXTERNAL_TABLE_AUTO_REFRESH
        • STORAGE.AUTO_CLUSTERING_SUSPEND
        • STORAGE.CHANGE_TRACKING
        • STORAGE.COLUMN_STORE
        • STORAGE.DYNAMIC_TABLE_BACKFILL_FROM
        • STORAGE.DYNAMIC_TABLE_IMMUTABLE_WHERE
        • STORAGE.DYNAMIC_TABLE_INITIALIZE_BEHAVIOR
        • STORAGE.DYNAMIC_TABLE_SIMPLE_ENTITY_RECREATE
        • STORAGE.EXTERNAL_TABLE_LOCATION
        • STORAGE.EXTERNAL_TABLE_FILE_FORMAT
        • STORAGE.EXTERNAL_TABLE_INTEGRATION
        • STORAGE.EXTERNAL_TABLE_FILENAME_PATTERN
        • STORAGE.EXTERNAL_TABLE_USER_SPEC_PARTITION
        • STORAGE.EXTERNAL_TABLE_REFRESH_ON_CREATE
        • STORAGE.EXTERNAL_TABLE_AWS_SNS_TOPIC
        • STORAGE.ICEBERG_EXTERNAL_LOCATION
        • STORAGE.ICEBERG_POINTER_FILE_STAGE
        • STORAGE.ICEBERG_SERIALIZATION_POLICY
        • STORAGE.PARTITION_SCHEME
        • STORAGE.REQUIRE_PARTITION_FILTER
        • STORAGE.SEARCH_OPTIMIZATION
        • STORAGE.SEARCH_OPTIMIZATION_OPTIONS
        • STORAGE.SECURE_VIEW
        • STORAGE.SORTKEY_RECLUSTER_IN_ENTITY_RECREATE
        • STORAGE.TABLE_STATISTICS_ATTRIBUTES
        • STORAGE.TABLE_STATISTICS_OPTIONS
        • STORAGE.TABLE_STATISTICS_UPDATE_IN_ENTITY_RECREATE
        • STORAGE.TABLE_STATISTICS_WHERE
        • STORAGE.TIME_TRAVEL_RETENTION_IN_DAYS
        • STORAGE.TRANSIENT_TABLE
      Loads
        Entity mapping
        • Entity mapping types
      • Import mappings
        • Load languages
        • SQL
          • API_CALL
          • Schema
        Load options
        • CUSTOM
        • OPT_DATA_FILE_LOCATION
        • OPT_DELTA_EXTRACT
        • OPT_FILE_FORMAT_OPTIONS
        • OPT_HISTORIZED_BY_DATAHASH_AS_INSERT
        • OPT_INSERT_UPDATE_BY_DATAHASH_AS_MERGE
        • OPT_INSERT_UPDATE_BY_DATAHASH_AS_OVERWRITE
        • OPT_INSERT_UPDATE_BY_DATAHASH_AS_SWAP
        • OPT_INSERT_UPDATE_BY_ID_AS_MERGE
        • OPT_INSERT_UPDATE_BY_ID_AS_OVERWRITE
        • OPT_INSERT_UPDATE_BY_ID_AS_SWAP
        • OPT_INSERT_UPDATE_DELETE_BY_DATAHASH_AS_MERGE
        • OPT_INSERT_UPDATE_DELETE_BY_DATAHASH_AS_OVERWRITE
        • OPT_INSERT_UPDATE_DELETE_BY_DATAHASH_AS_SWAP
        • OPT_INSERT_UPDATE_DELETE_BY_ID_AS_MERGE
        • OPT_INSERT_UPDATE_DELETE_BY_ID_AS_OVERWRITE
        • OPT_INSERT_UPDATE_DELETE_BY_ID_AS_SWAP
        • OPT_LOAD_POOL
        • OPT_LOAD_RETRY
        • OPT_LOAD_WITH_BQ_API
        • OPT_LOCK_TIMEOUT_IN_SECS
        • OPT_LOCK_WAIT_TIME_IN_SECS
        • OPT_NOT_EXISTS_FROM
        • OPT_NOT_EXISTS_WHERE
        • OPT_NUM_RUN_IDS_PER_TRANSACTION
        • OPT_ORDER_BY
        • OPT_SCAN_ONLY_THE_LATEST_BY_KEY
        • OPT_SCAN_ONLY_THE_LATEST_WITH_SOURCE_ATTRIBUTE_TYPE
        • OPT_SF_FILE_COPY_OPTIONS
        • OPT_SKIP_DISTINCT_ONLY
        • OPT_SKIP_DUPLICATE_PREVENTION
        • OPT_STORAGE_INTEGRATION
        • OPT_USE_RUN_IDS
        • OPT_USE_RUN_IDS_INCREMENTALLY
        • OPT_USE_TIMESLOT_LOGIC
        • OPT_WHERE
        Load step types
        • GATEKEEPER
        • OVERRIDE
        • OVERRIDE_FILE_LOAD
        • POST
        • PRE
        • SMOKE_BLACK
        • SMOKE_GREY
        Load types
        • LOAD_FILE
        • TRANSFORM_PERSIST
        • TRANSFORM_SQL_VIEW
      Variables
      • copy_options
      • dag_generation_mode
      • data_source
      • entity_lock_timeout_in_secs
      • entity_partition_key
      • entity_partition_key_{N}
      • env_short_name
      • environment_name
      • file_format
      • from_file
      • from_file_list
      • from_path
      • load_start_timemillis
      • loadablerunids
      • loadname
      • manifestrunid
      • multisource_mode
      • packagename
      • packageversion
      • scheduling_name
      • source_credentials
      • source_encryption
      • source_entity_attribute_by_{attribute_type}
      • source_entity_attribute_list
      • source_entity_name
      • source_entity_name_{N}
      • source_entity_schema
      • source_entity_schema_{N}
      • source_system_name
      • sourcebatchidattr (deprecated)
      • sourcerunidattr
      • stagingbatchidattr (deprecated)
      • storage_integration
      • system.deployment.entity_deletion.method
      • system.deployment.entity_deletion.method.{physical_type}
      • system.deployment.entity_recreate.dependent_dynamic_table.refresh
      • system.runtime.load_retry
      • target_db_name
      • target_entity_attribute_by_{attribute_type}
      • target_entity_attribute_list
      • target_entity_attribute_list_with_transform
      • target_entity_attribute_list_with_transform_cast_and_positions
      • target_entity_attribute_list_with_transform_cast_and_names
      • target_entity_attribute_list_with_transform_cast_exttable
      • target_entity_attribute_list_with_transform_cast_exttable_parquet
      • target_entity_attribute_list_with_transform_cast_parquet
      • target_entity_logical_name
      • target_entity_name
      • target_entity_transformation_by_{attribute_type}
      • target_schema
      • targetbatchid (deprecated)
      • targetbatchidattr (deprecated)
      • targetrunid
      • targetrunidattr
      • timemillis
      • warehouse_name
      Deployment Management
        Deployments UI
        • Designer main view
        • Deployments view
        • Package view
      Workflow Orchestration
      • DAG generation mode
      • Pools
      • System DAGs
      • Triggering a Workflow
      • Workflow Preconditions
      Insights
      • Access and Roles
      • DataOps Management KPIs
        • Insights Settings
        • Documentation Rules
        • Object Naming Rules
        • User and Tag Groups
      Notify API
        Manifests
        • Create manifest (POST)
        • Get manifest (GET)
        • Notify manifest (POST)
        • Search manifests (GET)
        Manifest entries
        • Create entry (POST)
        • Create multiple entries (PUT)
        • Get entries (GET)
        • Get entry (GET)
    • External API
      • Notification Events
      • AWS SNS
      • Azure Event Grid
      • Google Pub/Sub
      • Notification Types v1
    Release Notes
      Agile Data Engine 26.0
      • Release Notes 26.0
      • Release Notes 26.0.645
      Agile Data Engine 25.2
      • Release Notes 25.2
      • Release Notes 25.2.639
      • Release Notes 25.2.640
      • Release Notes 25.2.641
      • Release Notes 25.2.642
      • Release Notes 25.2.643
      Agile Data Engine 25.1
      • Release Notes 25.1.632.patch13
      • Release Notes 25.1.September.2
      • Release Notes 25.1.632.patch12
      • Release Notes 25.1.September.1
      • Release Notes 25.1.632.patch11
      • Release Notes 25.1.632.patch10
      • Release Notes 25.1.632.patch9
      • Release Notes 25.1.July.2
      • Release Notes 25.1.July.1
      • Release Notes 25.1.632.patch8
      • Release Notes 25.1.632.patch6
      • Release Notes 25.1.June.1
      • Release Notes 25.1.May.1
      • Release Notes 25.1.632.patch5
      • Release Notes 25.1.632.patch4
      • Release Notes 25.1.632.patch3
      • Release Notes 25.1.632.patch2
      • Release Notes 25.1.632.patch1
      • Release Notes 25.1.632
      • Release Notes 25.1
      Agile Data Engine 25.0
      • Release Notes 25.0.April.3
      • Release Notes 25.0.April.2
      • Release Notes 25.0.629
      • Release Notes 25.0.April.1
      • Release Notes 25.0.628
      • Release Notes 25.0.627.patch3
      • Release Notes 25.0.627.patch1
      • Release Notes 25.0
      Agile Data Engine 24.0
      • Release Notes 24.0.569
      • Release Notes 24.0.560
      • Release Notes 24.0.553
      • Release Notes 24.0
      Agile Data Engine 23.1
      • Release Notes 23.1.540
      • Release Notes 23.1.534
      • Release Notes 23.1.532
      • Release Notes 23.1
      Agile Data Engine 23.0
      • Release Notes 23.0.MARCH for External API
      • Release Notes 23.0.492.patch1
      • Release Notes 23.0
      Agile Data Engine 22.1
      • Release Notes 22.1.11.patch19
      • Release Notes 22.1.11.patch16
      • Release Notes 22.1.11.patch15
      • Release Notes 22.1.11.patch14
      • Release Notes 22.1.11.patch13
      • Release Notes 22.1.11.patch12
      • Release Notes 22.1.11.patch11
      • Release Notes 22.1.11.patch10
      • Release Notes 22.1.11.patch9
      • Release Notes 22.1.11.patch8
      • Release Notes 22.1.11.patch7
      • Release Notes 22.1.11.patch6
      • Release Notes 22.1.11.patch5
      • Release Notes 22.1.11.patch4
      • Release Notes 22.1.11.patch3
      • Release Notes 22.1.11.patch2
      • Release Notes 22.1.11.patch1
      • Release Notes 22.1.11
      • Release Notes 22.1.10
      • Release Notes 22.1
      Agile Data Engine 22.0
      • Release Notes 22.0.469
      • Release Notes 22.0.468
      • Release Notes 22.0.467.patch1
      • Release Notes 22.0.467
      • Release Notes 22.0.466
      • Release Notes 22.0.464
      • Release Notes 22.0.460
      • Release Notes 22.0
      Agile Data Engine 21.0
      • Release Notes 21.0.443.patch25
      • Release Notes 21.0.443.patch24
      • Release Notes 21.0.443.patch23
      • Release Notes 21.0.443.patch22
      • Release Notes 21.0.443.patch21
      • Release Notes 21.0.443.patch20
      • Release Notes 21.0.443.patch19
      • Release Notes 21.0.443.patch18
      • Release Notes 21.0.443.patch17
      • Release Notes 21.0.443.patch16
      • Release Notes 21.0.443.patch15
      • Release Notes 21.0.443.patch14
      • Release Notes 21.0.443.patch13
      • Release Notes 21.0.443.patch12
      • Release Notes 21.0.443.patch11
      • Release Notes 21.0.443.patch10
      • Release Notes 21.0.443.patch9
      • Release Notes 21.0.443.patch8
      • Release Notes 21.0.443.patch7
      • Release Notes 21.0.443.patch6
      • Release Notes 21.0.443.patch5
      • Release Notes 21.0.443.patch4
      • Release Notes 21.0.443.patch3
      • Release Notes 21.0.443.patch2
      • Release Notes 21.0.443.patch1
      • Release Notes 21.0.443
      • Release Notes 21.0.435.patch18
      • Release Notes 21.0.435.patch17
      • Release Notes 21.0.435.patch16
      • Release Notes 21.0.435.patch15
      • Release Notes 21.0.435.patch14
      • Release Notes 21.0.435.patch13
      • Release Notes 21.0.435.patch12
      • Release Notes 21.0.435.patch11
      • Release Notes 21.0.435.patch10
      • Release Notes 21.0.435.patch9
      • Release Notes 21.0.435.patch8
      • Release Notes 21.0.435.patch7
      • Release Notes 21.0.435.patch6
      • Release Notes 21.0.435.patch5
      • Release Notes 21.0.435.patch4
      • Release Notes 21.0.435.patch3
      • Release Notes 21.0.435.patch2
      • Release Notes 21.0.435.patch1
      • Release Notes 21.0.435
      • Release Notes 21.0.420.patch8
      • Release Notes 21.0.420.patch7
      • Release Notes 21.0.420.patch6
      • Release Notes 21.0.420.patch5
      • Release Notes 21.0.420.patch4
      • Release Notes 21.0.420.patch3
      • Release Notes 21.0.420.patch2
      • Release Notes 21.0.420.patch1
      • Release Notes 21.0.420
      • Release Notes 21.0
      Agile Data Engine 20.2
      • Release Notes 20.2.396.patch11
      • Release Notes 20.2.396.patch10
      • Release Notes 20.2.396.patch9
      • Release Notes 20.2.2.396.patch8
      • Release Notes 20.2
      Agile Data Engine 20.1
      • Release Notes 20.1
      Agile Data Engine 20.0
      • Release Notes 20.0

Agile Data Engine Documentation

    What is Agile Data Engine

    • Key Concepts and Architecture

    Getting Started

    • Tenant and Environments
    • Accessing ADE
    • Configuring ADE
    • Importing Initial Metadata
    • Setting Up Data Ingestion
    • API Access and Setup

    Configuration Guide

    • Admin UI
    • Configuration Packages
    • Microsoft Entra ID Authentication Integration
    • Private Connectivity and VPN
    • Role-Based Access Control (RBAC)
    • Target Database Setup

    User Guide

    • Data Warehouse Modeling
    • Solution examples
    • Designing Packages
    • Designing Workflows
    • From Metadata to DDL
    • Loading Source Data
    • Setting Up a File Notifier Application
    • Using Metadata Interface
    • Using Run ID Logic
    • FAQ

    Reference Guide

    • Packages
    • Entities
    • Loads
    • Variables
    • Deployment Management
    • Workflow Orchestration
    • Insights
    • Notify API
    • External API
    • Notification Events

    Release Notes

    • Agile Data Engine 26.0
    • Agile Data Engine 25.2
    • Agile Data Engine 25.1
    • Agile Data Engine 25.0
    • Agile Data Engine 24.0
    • Agile Data Engine 23.1
    • Agile Data Engine 23.0
    • Agile Data Engine 22.1
    • Agile Data Engine 22.0
    • Agile Data Engine 21.0
    • Agile Data Engine 20.2
    • Agile Data Engine 20.1
    • Agile Data Engine 20.0
  • LinkedIn
  • Facebook
  • Twitter
  • YouTube
  • Copyright Agile Data Engine
  • Powered by Scroll Sites & Atlassian Confluence