Databricks SQL

Agile Data Engine supports:

Databricks SQL

Required connection details

The following details are required per target instance configuration:

Detail	Example	Description
Host	`adb-1234567890123456.7.azuredatabricks.net`	Databricks workspace URL.
Port	`443`	Default port for Databricks SQL connections.
Catalog name*	`clouddw_dev`	Catalog (i.e. top-level “database”) name in Unity Catalog. It represents the namespace under which schemas, tables, views, and other objects are organized in Databricks.
Default schema	`PUBLIC`	Default schema within the Unity Catalog.
Warehouse HTTP path	`/sql/1.0/warehouses/abcdefghijklmnopqrstuvwxyz`	HTTP endpoint for your Databricks SQL warehouse. Obtain this value under SQL Warehouses → Connection details in your workspace UI.
Microsoft Entra ID service principal (recommended)	Client ID: `12a34b56-789c-0d12-e3fa-b456789c0123` Tenant ID: `a1bc2d34-5e67-8f89-01ab-c2345c6d78de` Client secret: `XyZ1234567890abcdef`	Authenticate using a Microsoft Entra ID service principal. Recommended authentication method for production use.
Databricks service principal details (alternative)	Client ID: `1234-5678-9012-3456` Client secret: `XyZ1234567890abcdef`	Authenticate using a Databricks-managed service principal. Recommended authentication method for production use when Microsoft Entra ID is not used.
Personal access token (alternative)	`dapi1234abcde5678fghij90klmnopqr`	A generated token tied to a Databricks user account. Not recommended for production use.

*Agile Data Engine requires Unity Catalog.

Configuration

To ensure proper isolation between environments (e.g. DEV, TEST, PROD), follow these best practices:

Create separate service principals per environment
This ensures independent authentication and access control scopes across environments.
Create separate catalogs per environment
For example: clouddw_dev, clouddw_test, clouddw_prod. This enables logical and permission-based separation of data assets.
Create separate SQL Warehouses per environment
For example ade_wh_dev, ade_wh_prod. Control size, auto-stop, and other settings, and track costs at the environment level.
Recommended: Create a separate Databricks Workspace per environment
While Unity Catalog enables soft isolation within a single workspace, separate workspaces provide stronger boundaries and simplify environment-specific configuration, access, and audit logging.

Service principals

To authenticate Agile Data Engine with Databricks SQL, you must create credentials. Choose one of the following methods, depending on your environment and security requirements:

Option A: Microsoft Entra ID service principal – Recommended

Create a service principal in Microsoft Entra ID, either via:

Azure Portal, or
Azure CLI, PowerShell, or Terraform

Save the following values:

Client ID
Tenant ID
Client Secret

This method is recommended for production setups.

Option B: Databricks-managed service principal – Alternative

Create a service principal directly within the Databricks workspace UI:

Go to Settings → Identity and access → Service principals
Click Add service principal

Save the following values:

Client ID
Client Secret

This method is suitable for Databricks-only use cases or when Microsoft Entra ID is not available.

Option C: Personal access token (PAT) – For testing only

For proof-of-concept (PoC) work and quick testing:

Navigate to Settings → Developer → Access tokens
Click Generate new token

Save the generated:

Token string

This method is not recommended for production use.

External locations for loading source data

To enable ADE to orchestrate file loads from your cloud storage via Unity Catalog in Databricks, configure external locations in Databricks. See instructions in Databricks documentation.