Breadcrumbs

Databricks SQL

Agile Data Engine supports:

  • Databricks SQL

Required connection details

The following details are required per target instance configuration:

Detail

Example

Description

Host

adb-1234567890123456.7.azuredatabricks.net

Databricks workspace URL.

Port

443

Default port for Databricks SQL connections.

Catalog name*

clouddw_dev

Catalog (i.e. top-level “database”) name in Unity Catalog. It represents the namespace under which schemas, tables, views, and other objects are organized in Databricks.

Default schema

PUBLIC

Default schema within the Unity Catalog.

Warehouse HTTP path

/sql/1.0/warehouses/abcdefghijklmnopqrstuvwxyz

HTTP endpoint for your Databricks SQL warehouse. Obtain this value under SQL Warehouses → Connection details in your workspace UI.

Microsoft Entra ID service principal (recommended)

  • Client ID:
    12a34b56-789c-0d12-e3fa-b456789c0123

  • Tenant ID:
    a1bc2d34-5e67-8f89-01ab-c2345c6d78de

  • Client secret:
    XyZ1234567890abcdef

Authenticate using a Microsoft Entra ID service principal.

Recommended authentication method for production use.

Databricks service principal details
(alternative)

  • Client ID:
    1234-5678-9012-3456

  • Client secret:
    XyZ1234567890abcdef

Authenticate using a Databricks-managed service principal.

Recommended authentication method for production use when Microsoft Entra ID is not used.

Personal access token
(alternative)

dapi1234abcde5678fghij90klmnopqr

A generated token tied to a Databricks user account.

Not recommended for production use.

*Agile Data Engine requires Unity Catalog.

Configuration

To ensure proper isolation between environments (e.g. DEV, TEST, PROD), follow these best practices:

  • Create separate service principals per environment
    This ensures independent authentication and access control scopes across environments.

  • Create separate catalogs per environment
    For example: clouddw_dev, clouddw_test, clouddw_prod. This enables logical and permission-based separation of data assets.

  • Create separate SQL Warehouses per environment
    For example ade_wh_dev, ade_wh_prod. Control size, auto-stop, and other settings, and track costs at the environment level.

  • Recommended: Create a separate Databricks Workspace per environment
    While Unity Catalog enables soft isolation within a single workspace, separate workspaces provide stronger boundaries and simplify environment-specific configuration, access, and audit logging.

Service principals

To authenticate Agile Data Engine with Databricks SQL, you must create credentials. Choose one of the following methods, depending on your environment and security requirements:

Option A: Microsoft Entra ID service principal

Authentication type: DATABRICKS_ENTRA_ID_SERVICE_PRINCIPAL

Create a service principal in Microsoft Entra ID, either via:

  • Azure Portal, or

  • Azure CLI, PowerShell, or Terraform

Save the following values:

  • Client ID

  • Tenant ID

  • Client Secret

This method is recommended for Azure Databricks when using Entra ID for identity management.

Authentication type: DATABRICKS_SERVICE_PRINCIPAL_OAUTH_M2M

Create a service principal directly within the Databricks workspace UI:

  1. Go to Settings → Identity and access → Service principals

  2. Click Add service principal

Save the following values:

  • Client ID

  • Client Secret

This method uses native OAuth tokens managed automatically by the Databricks SDK/driver. No PAT permissions required for the service principal. Recommended for new setups.

Option C: Databricks service principal with PAT generation – Alternative

Authentication type: DATABRICKS_SERVICE_PRINCIPAL

Same setup as Option B, but ADE generates short-lived Personal Access Tokens (PATs) via the Databricks API.

Use this method for backwards compatibility or when OAuth M2M is not available.

Option D: Personal access token (PAT) – For testing only

Authentication type: DATABRICKS_PERSONAL_ACCESS_TOKEN

For proof-of-concept (PoC) work and quick testing:

  • Navigate to Settings → Developer → Access tokens

  • Click Generate new token

Save the generated:

  • Token string

This method is not recommended for production use.

External locations for loading source data

To enable ADE to orchestrate file loads from your cloud storage via Unity Catalog in Databricks, configure external locations in Databricks. See instructions in Databricks documentation.