CONFIG_DATATYPES

CONFIG_DATATYPES is a configuration package used to define how importable (physical) data types from source systems are mapped to logical data types within Agile Data Engine during entity metadata import. It also defines how those logical types are translated into physical data types in target databases during deployment.

In this context:

Importable data types are the concrete physical data types found in source systems (e.g., VARCHAR, INT, DECIMAL). These are used during entity import.
Logical data types are abstract representations understood by Agile Data Engine.
Physical data types (target) are the actual data types used in the deployed target databases.

The purpose of this mapping is to ensure consistency and portability. During development, attributes are assigned logical data types. Then, during deployment, these logical types are automatically converted into appropriate physical types for the chosen target system.

For example:

A source system like Microsoft SQL Server might use the physical type DATETIME2. This can be mapped to the logical type TIMESTAMP, which is then deployed as TIMESTAMP_NTZ in Databricks SQL or Snowflake.

This abstraction layer enables teams to build database-agnostic models while still delivering optimized, platform-specific implementations.

See also:

Usage

To manage data type mappings, open the CONFIG_DATATYPES package and navigate to the Data Types tab. From there, you can add, modify, or remove the following:

Logical data types
Physical data types
Importable data types

Alternatively, you can edit the package directly by opening Show Editor, or by exporting the JSON file, making changes externally, and then re-importing it. See the Contents section below for reference.

After saving changes, the CONFIG_DATATYPES package will contain uncommitted changes. To preserve change history, it is recommended to commit the package. For changes to take effect, be sure to deploy them to the appropriate Runtime environments.

Key	Value type	Example	Description
logicalDatatype	String	INTEGER8	Logical type name.
usesDataLength	Boolean	false	Specifies whether the logical data type uses data length. Length refers to the maximum number of characters or bytes allowed for values of this data type. For example, a logical type that uses length might map to `VARCHAR(255)` in a physical database, where `255` is the defined maximum length of the value.
usesPrecisionMetadata	Boolean	false	Specifies whether the logical data type uses precision. Precision defines the total number of digits that can be stored for numeric values. For example, a logical type with precision set to `10` might map to `DECIMAL(10, x)` in a physical database, where `10` is the total number of digits allowed.
usesScale	Boolean	false	Specifies whether the logical data type uses scale. Scale defines the number of digits allowed after the decimal point for numeric values. For example, a logical type with scale `2` may map to `DECIMAL(x, 2)` in a physical database, where `2` specifies that two digits are stored after the decimal point.

Example: Logical data type configuration

JSON

"logicalDataTypes": [
  ...
  {
    "logicalDatatype": "INTEGER8",
    "usesDataLength": false,
    "usesPrecisionMetadata": false,
    "usesScale": false
  }
  ...
]

Physical data type configuration

Physical data type configurations are managed within the JSON array block named physicalDatatypes.

Key	Value type	Example	Description
physicalDatatypeId	String	a9107d12-1ada-0794-bd93-43a3d8172c83	Unique identifier for the physical data type. Can be generated with online tools, such as UUID Generator.
dbmsProduct	String	DATABRICKS	DBMS product refers to the database management system for which the physical data type mapping applies.
physicalDatatype	String	TIMESTAMP_NTZ	Physical data type refers to the actual data type used in a database system. These types are specific to each DBMS and determine how data is stored and interpreted at the database level.
logicalDatatype	String	TIMESTAMP	Reference to the logical data type configured within `logicalDataTypes`.
usesDataLength	Boolean	false	Specifies whether the physical data type uses data length. Length refers to the maximum number of characters or bytes allowed for values of this data type, e.g. VARCHAR(255).
usesPrecisionMetadata	Boolean	false	Specifies whether the physical data type uses precision. Precision defines the total number of digits that can be stored for numeric values, e.g. DECIMAL(10,x).
usesScale	Boolean	false	Specifies whether the physical data type uses scale. Scale defines the number of digits allowed after the decimal point for numeric values, e.g. DECIMAL(x,2).

Example: Physical data type configuration

JSON

"dwPhysicalDatatypes": [
  ...
  {
    "physicalDatatypeId": "a9107d12-1ada-0794-bd93-43a3d8172c83",
    "dbmsProduct": "DATABRICKS",
    "physicalDatatype": "TIMESTAMP_NTZ",
    "logicalDatatype": "TIMESTAMP"
    "usesDataLength": false,
    "usesPrecisionMetadata": false,
    "usesScale": false
  }
  ...
]

Importable data type configuration

Importable data type configurations are managed within the JSON array block named dwPhysicalDatatypeToLogicals.

Key	Value type	Example	Description
physicalDatatypeToLogicalId	String	2e7f77cc-4150-2ad8-c276-1d9a012a3940	Unique identifier for the importable data type. Can be generated with online tools, such as UUID Generator.
dbmsProduct	String	DUCKDB	DBMS product refers to the database management system for which the importable data type mapping applies.
physicalDatatype	String	HUGEINT	Physical data type refers to the actual data type used in a database system. These types are specific to each DBMS and determine how data is stored and interpreted at the database level.
logicalDatatype	String	VARCHAR	Reference to the logical data type configured within `logicalDataTypes`. This field defines which logical data type the importable data type maps to during entity attribute metadata import.
defaultPrecision	Integer	null	Optional: Specifies the default precision for the logical data type during import, if applicable.
defaultScale	Integer	null	Optional: Specifies the default scale for the logical data type during import, if applicable.
defaultLength	Integer	40	Optional: Specifies the default length for the logical data type during import, if applicable.

Example: Importable data type configuration

JSON

"importableDatatypes": [
  ...
  {
    "physicalDatatypeToLogicalId": "2e7f77cc-4150-2ad8-c276-1d9a012a3940",
    "dbmsProduct": "DUCKDB",
    "physicalDatatype": "HUGEINT",
    "logicalDatatype": "VARCHAR"
    "defaultPrecision": null,
    "defaultScale": null,
    "defaultLength": 40
  }
  ...
]