Skip to main content

Main

Main

What is Agile Data Engine

Key Concepts and Architecture

Getting Started

Tenant and Environments
Accessing ADE
Configuring ADE
Importing Initial Metadata

Setting Up Data Ingestion

Getting started with Notify API
Data Ingestion with External Tools

API Access and Setup

Configuration Guide

Admin UI

Configuration Packages

CONFIG_AUTOCOMPLETIONS

CONFIG_DATATYPES

INLINE DATATYPE

CONFIG_ENTITY_DEFAULTS
CONFIG_ENTITY_PERMISSIONS
CONFIG_ENVIRONMENT_VARIABLES
CONFIG_ISSUE_TRACK_SYSTEM
CONFIG_LOAD_SCHEDULES

CONFIG_LOAD_TEMPLATES

FTL Template Reference

Product-defined FTL templates

Supported FTL functions and macros

loads (provided FTL library)
sql (provided FTL library)
utils (provided FTL library)

CONFIG_LOAD_TRANSFORMATIONS
CONFIG_SYSTEMS
CONFIG_VALIDATIONS

Microsoft Entra ID Authentication Integration

Microsoft Entra ID Authentication Integration (User Management v1)

Private Connectivity and VPN
Role-Based Access Control (RBAC)

Target Database Setup

Amazon Redshift
Azure SQL Database
Azure Synapse SQL
Databricks SQL
Google BigQuery
Microsoft Fabric
Snowflake

User Guide

Data Warehouse Modeling

Data Vault

Hub
Link
Satellite
Status satellite
Multi-active satellite
Point-in-Time (PIT)
Transactional link

Dimensional

Fact

Dimension

SCD Type 2 with FTL Templates

Persistent staging
Customizing default attributes and transformations

Solution examples

Customizing File Loads in Snowflake
Dynamic tables in Snowflake
Hybrid table support in Snowflake
Load steps with user-defined functions
Sensitive Data Vault with FTL Templates
Staging with streaming tables in Databricks
Unloading data to cloud storage
Using Google Cloud services with API_CALL
Working with data shares

Designing Packages
Designing Workflows
From Metadata to DDL
Loading Source Data
Setting Up a File Notifier Application
Using Metadata Interface
Using Run ID Logic
FAQ

Reference Guide

Packages

Commit Package
Export Packages
Import Package
Package Actions
Package Change Comparison
Package Dependencies

Entities

Clone entity
Import entities
Move to another package
Entity types

Attributes

COLLATION VALUE
MASKING POLICY
TECHNICAL ATTRIBUTE TYPE

Database Tags

Configuration of Database Tags

Distribution styles

ALL
EVEN
KEY

Keys & References

DISTRIBUTION_KEY
FOREIGN_KEY
INTERLEAVED_SORT_KEY
PARTITION_KEY
PRIMARY_KEY
SEARCH_INDEX_KEY
SECONDARY_INDEX
SORT_KEY
UNIQUE_KEY

Permissions

Row-level permissions

Configuration of Row-level permissions

Physical types

DYNAMIC_TABLE
EXTERNAL_TABLE
ICEBERG_TABLE
MATERIALIZED_VIEW
METADATA_ONLY
SEMANTIC_VIEW
TABLE
VIEW

Physical options

DEPLOYMENT.COMPUTE_CLUSTER
REFRESH.DYNAMIC_TABLE_REFRESH_MODE
REFRESH.DYNAMIC_TABLE_TARGET_LAG
REFRESH.DYNAMIC_TABLE_WAREHOUSE
REFRESH.EXTERNAL_TABLE_AUTO_REFRESH
STORAGE.AUTO_CLUSTERING_SUSPEND
STORAGE.CHANGE_TRACKING
STORAGE.COLUMN_STORE
STORAGE.DYNAMIC_TABLE_BACKFILL_FROM
STORAGE.DYNAMIC_TABLE_IMMUTABLE_WHERE
STORAGE.DYNAMIC_TABLE_INITIALIZE_BEHAVIOR
STORAGE.DYNAMIC_TABLE_SIMPLE_ENTITY_RECREATE
STORAGE.EXTERNAL_TABLE_LOCATION
STORAGE.EXTERNAL_TABLE_FILE_FORMAT
STORAGE.EXTERNAL_TABLE_INTEGRATION
STORAGE.EXTERNAL_TABLE_FILENAME_PATTERN
STORAGE.EXTERNAL_TABLE_USER_SPEC_PARTITION
STORAGE.EXTERNAL_TABLE_REFRESH_ON_CREATE
STORAGE.EXTERNAL_TABLE_AWS_SNS_TOPIC
STORAGE.ICEBERG_EXTERNAL_LOCATION
STORAGE.ICEBERG_POINTER_FILE_STAGE
STORAGE.ICEBERG_SERIALIZATION_POLICY
STORAGE.PARTITION_SCHEME
STORAGE.REQUIRE_PARTITION_FILTER
STORAGE.SEARCH_OPTIMIZATION
STORAGE.SEARCH_OPTIMIZATION_OPTIONS
STORAGE.SECURE_VIEW
STORAGE.SORTKEY_RECLUSTER_IN_ENTITY_RECREATE
STORAGE.TABLE_STATISTICS_ATTRIBUTES
STORAGE.TABLE_STATISTICS_OPTIONS
STORAGE.TABLE_STATISTICS_UPDATE_IN_ENTITY_RECREATE
STORAGE.TABLE_STATISTICS_WHERE
STORAGE.TIME_TRAVEL_RETENTION_IN_DAYS
STORAGE.TRANSIENT_TABLE
STORAGE.CLUSTERING_SCHEME
STORAGE.EXTERNAL_TABLE_CREDENTIALS

Loads

Entity mapping

Entity mapping types

Import mappings

Load languages

SQL

API_CALL

Schema

Load options

CUSTOM
OPT_DATA_FILE_LOCATION
OPT_DELTA_EXTRACT
OPT_FILE_FORMAT_OPTIONS
OPT_HISTORIZED_BY_DATAHASH_AS_INSERT
OPT_INSERT_UPDATE_BY_DATAHASH_AS_MERGE
OPT_INSERT_UPDATE_BY_DATAHASH_AS_OVERWRITE
OPT_INSERT_UPDATE_BY_DATAHASH_AS_SWAP
OPT_INSERT_UPDATE_BY_ID_AS_MERGE
OPT_INSERT_UPDATE_BY_ID_AS_OVERWRITE
OPT_INSERT_UPDATE_BY_ID_AS_SWAP
OPT_INSERT_UPDATE_DELETE_BY_DATAHASH_AS_MERGE
OPT_INSERT_UPDATE_DELETE_BY_DATAHASH_AS_OVERWRITE
OPT_INSERT_UPDATE_DELETE_BY_DATAHASH_AS_SWAP
OPT_INSERT_UPDATE_DELETE_BY_ID_AS_MERGE
OPT_INSERT_UPDATE_DELETE_BY_ID_AS_OVERWRITE
OPT_INSERT_UPDATE_DELETE_BY_ID_AS_SWAP
OPT_LOAD_POOL
OPT_LOAD_RETRY
OPT_LOAD_WITH_BQ_API
OPT_LOCK_TIMEOUT_IN_SECS
OPT_LOCK_WAIT_TIME_IN_SECS
OPT_NOT_EXISTS_FROM
OPT_NOT_EXISTS_WHERE
OPT_NUM_RUN_IDS_PER_TRANSACTION
OPT_ORDER_BY
OPT_SCAN_ONLY_THE_LATEST_BY_KEY
OPT_SCAN_ONLY_THE_LATEST_WITH_SOURCE_ATTRIBUTE_TYPE
OPT_SF_FILE_COPY_OPTIONS
OPT_SKIP_DISTINCT_ONLY
OPT_SKIP_DUPLICATE_PREVENTION
OPT_STORAGE_INTEGRATION
OPT_USE_RUN_IDS
OPT_USE_RUN_IDS_INCREMENTALLY
OPT_USE_TIMESLOT_LOGIC
OPT_WHERE

Load step types

GATEKEEPER
OVERRIDE
OVERRIDE_FILE_LOAD
POST
PRE
SMOKE_BLACK
SMOKE_GREY

Load types

LOAD_FILE
TRANSFORM_PERSIST
TRANSFORM_SQL_VIEW

Variables

copy_options
dag_generation_mode
data_source
entity_clustering_key
entity_clustering_key_{N}
entity_lock_timeout_in_secs
entity_partition_key
entity_partition_key_{N}
env_short_name
environment_name
file_format
from_file
from_file_list
from_path
load_start_timemillis
loadablerunids
loadname
manifestrunid
multisource_mode
packagename
packageversion
scheduling_name
source_credentials
source_encryption
source_entity_attribute_by_{attribute_type}
source_entity_attribute_list
source_entity_name
source_entity_name_{N}
source_entity_schema
source_entity_schema_{N}
source_system_name
sourcebatchidattr (deprecated)
sourcerunidattr
stagingbatchidattr (deprecated)
storage_integration
system.deployment.entity_deletion.method
system.deployment.entity_deletion.method.{physical_type}
system.deployment.entity_recreate.dependent_dynamic_table.refresh
system.runtime.load_retry
target_db_name
target_entity_attribute_by_{attribute_type}
target_entity_attribute_list
target_entity_attribute_list_with_transform
target_entity_attribute_list_with_transform_cast_and_positions
target_entity_attribute_list_with_transform_cast_and_names
target_entity_attribute_list_with_transform_cast_exttable
target_entity_attribute_list_with_transform_cast_exttable_parquet
target_entity_attribute_list_with_transform_cast_parquet
target_entity_logical_name
target_entity_name
target_entity_transformation_by_{attribute_type}
target_schema
targetbatchid (deprecated)
targetbatchidattr (deprecated)
targetrunid
targetrunidattr
timemillis
warehouse_name

Deployment Management

Deployments UI

Designer main view
Deployments view
Package view

Workflow Orchestration

DAG generation mode
Pools
System DAGs
Triggering a Workflow
Workflow Preconditions

Insights

Access and Roles
DataOps Management KPIs

Insights Settings

Documentation Rules
Object Naming Rules
User and Tag Groups

Notify API

Manifests

Create manifest (POST)
Get manifest (GET)
Notify manifest (POST)
Search manifests (GET)

Manifest entries

Create entry (POST)
Create multiple entries (PUT)
Get entries (GET)
Get entry (GET)

External API

Notification Events

AWS SNS
Azure Event Grid
Google Pub/Sub
Notification Types v1

Release Notes

Agile Data Engine 26.0

Release Notes 26.0
Release Notes 26.0.645
Release Notes 26.0.647

Agile Data Engine 25.2

Release Notes 25.2
Release Notes 25.2.639
Release Notes 25.2.640
Release Notes 25.2.641
Release Notes 25.2.642
Release Notes 25.2.643

Agile Data Engine 25.1

Release Notes 25.1.632.patch13
Release Notes 25.1.September.2
Release Notes 25.1.632.patch12
Release Notes 25.1.September.1
Release Notes 25.1.632.patch11
Release Notes 25.1.632.patch10
Release Notes 25.1.632.patch9
Release Notes 25.1.July.2
Release Notes 25.1.July.1
Release Notes 25.1.632.patch8
Release Notes 25.1.632.patch6
Release Notes 25.1.June.1
Release Notes 25.1.May.1
Release Notes 25.1.632.patch5
Release Notes 25.1.632.patch4
Release Notes 25.1.632.patch3
Release Notes 25.1.632.patch2
Release Notes 25.1.632.patch1
Release Notes 25.1.632
Release Notes 25.1

Agile Data Engine 25.0

Release Notes 25.0.April.3
Release Notes 25.0.April.2
Release Notes 25.0.629
Release Notes 25.0.April.1
Release Notes 25.0.628
Release Notes 25.0.627.patch3
Release Notes 25.0.627.patch1
Release Notes 25.0

Agile Data Engine 24.0

Release Notes 24.0.569
Release Notes 24.0.560
Release Notes 24.0.553
Release Notes 24.0

Agile Data Engine 23.1

Release Notes 23.1.540
Release Notes 23.1.534
Release Notes 23.1.532
Release Notes 23.1

Agile Data Engine 23.0

Release Notes 23.0.MARCH for External API
Release Notes 23.0.492.patch1
Release Notes 23.0

Agile Data Engine 22.1

Release Notes 22.1.11.patch19
Release Notes 22.1.11.patch16
Release Notes 22.1.11.patch15
Release Notes 22.1.11.patch14
Release Notes 22.1.11.patch13
Release Notes 22.1.11.patch12
Release Notes 22.1.11.patch11
Release Notes 22.1.11.patch10
Release Notes 22.1.11.patch9
Release Notes 22.1.11.patch8
Release Notes 22.1.11.patch7
Release Notes 22.1.11.patch6
Release Notes 22.1.11.patch5
Release Notes 22.1.11.patch4
Release Notes 22.1.11.patch3
Release Notes 22.1.11.patch2
Release Notes 22.1.11.patch1
Release Notes 22.1.11
Release Notes 22.1.10
Release Notes 22.1

Agile Data Engine 22.0

Release Notes 22.0.469
Release Notes 22.0.468
Release Notes 22.0.467.patch1
Release Notes 22.0.467
Release Notes 22.0.466
Release Notes 22.0.464
Release Notes 22.0.460
Release Notes 22.0

Agile Data Engine 21.0

Release Notes 21.0.443.patch25
Release Notes 21.0.443.patch24
Release Notes 21.0.443.patch23
Release Notes 21.0.443.patch22
Release Notes 21.0.443.patch21
Release Notes 21.0.443.patch20
Release Notes 21.0.443.patch19
Release Notes 21.0.443.patch18
Release Notes 21.0.443.patch17
Release Notes 21.0.443.patch16
Release Notes 21.0.443.patch15
Release Notes 21.0.443.patch14
Release Notes 21.0.443.patch13
Release Notes 21.0.443.patch12
Release Notes 21.0.443.patch11
Release Notes 21.0.443.patch10
Release Notes 21.0.443.patch9
Release Notes 21.0.443.patch8
Release Notes 21.0.443.patch7
Release Notes 21.0.443.patch6
Release Notes 21.0.443.patch5
Release Notes 21.0.443.patch4
Release Notes 21.0.443.patch3
Release Notes 21.0.443.patch2
Release Notes 21.0.443.patch1
Release Notes 21.0.443
Release Notes 21.0.435.patch18
Release Notes 21.0.435.patch17
Release Notes 21.0.435.patch16
Release Notes 21.0.435.patch15
Release Notes 21.0.435.patch14
Release Notes 21.0.435.patch13
Release Notes 21.0.435.patch12
Release Notes 21.0.435.patch11
Release Notes 21.0.435.patch10
Release Notes 21.0.435.patch9
Release Notes 21.0.435.patch8
Release Notes 21.0.435.patch7
Release Notes 21.0.435.patch6
Release Notes 21.0.435.patch5
Release Notes 21.0.435.patch4
Release Notes 21.0.435.patch3
Release Notes 21.0.435.patch2
Release Notes 21.0.435.patch1
Release Notes 21.0.435
Release Notes 21.0.420.patch8
Release Notes 21.0.420.patch7
Release Notes 21.0.420.patch6
Release Notes 21.0.420.patch5
Release Notes 21.0.420.patch4
Release Notes 21.0.420.patch3
Release Notes 21.0.420.patch2
Release Notes 21.0.420.patch1
Release Notes 21.0.420
Release Notes 21.0

Agile Data Engine 20.2

Release Notes 20.2.396.patch11
Release Notes 20.2.396.patch10
Release Notes 20.2.396.patch9
Release Notes 20.2.2.396.patch8
Release Notes 20.2

Agile Data Engine 20.1

Release Notes 20.1

Agile Data Engine 20.0

Release Notes 20.0

Breadcrumbs

Configuration Guide
Configuration Packages
CONFIG_LOAD_TEMPLATES
FTL Template Reference

On this Page

Supported FTL functions and macros

loads (provided FTL library)
sql (provided FTL library)
utils (provided FTL library)

LinkedIn
Facebook
Twitter
YouTube

Copyright Agile Data Engine
Powered by Scroll Sites & Atlassian Confluence