Release Notes 24.0

1. RELEASE INFO

2. CONTENT OF THE RELEASE

2.1 Highlights

• Databricks SQL support advanced from Private Preview to Public Preview

  • Service Principal based Authentication Support in addition to already supported personal access tokens (PAT)

  • Databricks SQL on AWS is supported in addition to already supported Databricks SQL on Azure

• New Database Tags feature supporting TAG in Snowflake & Databricks SQL and RESOURCE TAG + POLICY TAG in Google BigQuery

• New Row Level Permissions feature

• Ease of use UIs for selected configurations (Config Packages)

• New SCD2 load option to support Slowly Changing Dimension Type 2 in Private Preview

• AI based Hashkey Suggestion in Private Preview

• New Business Key Automation (Copy from Hashkey) based on new BUSINESS_KEY technical attribute type

• New Always On Deployment Indicators in Designer

• Improved DataOps Management KPI dashboards in INSIGHTS with hashtag and package filters (SaaS Editions)

• Improved Private Connectivity Features in Business Critical Edition (SaaS Editions)

• Platform’s performance improvement of using 15% less CPU for the same workload (SaaS Editions)

2.2 User Experience Improvements

2.2.1 Designer

• Ease of use UIs for selected Config Packages

  • Entity Defaults

  • Environment Variables

  • Load Transformations

  • Issue Track Systems

  • Datatypes

• Always on Deployment Indicators

• Import/Export performance improved

• Code preview initial connection error fixed

• Load mapping support features

  • AI based Hashkey and Referencing Hashkey suggestion (when no Copy by ref. appears)

  • Business Key Automation (Copy from Hashkey) based on new BUSINESS_KEY technical attribute type

2.3 Deployment Management

• Fixed: Deployments UI’s too high CPU usage

• Log Snowflake query id always in Deployment and Workflow Orchestration logs

2.4 Loads

• New SCD2 load option to support Slowly Changing Dimension Type 2 in Private Preview

• Log Snowflake query id always in Deployment and Workflow Orchestration logs

• Update of Apache Airflow Version

2.5 Automatic schema management

• Support of Database Tags

• Support of Row Level Permissions

• Fixed: Google BigQuery partition day partitioning

• Added support for Snowflake datatypes: TIMESTAMP_LZ and GEOMETRY

• Log Snowflake query id always in Deployment and Workflow Orchestration logs

2.6 APIs

• External API: Update timestamp filtering support added to the Metadata Interface where possible

• Notify API: Performance improvements
  

3. NOTIFICATIONS

3.1 Deprecated features

3.1.1 Grafana alerts are deprecated

Grafana alerts will be inactivated from the customers not currently using them in SaaS edition as part of this release. For the current SaaS edition users of Grafana alerts there will be grace period until end of August 2024 for stopping of using them. Replacing feature is Notification Events.

For Private Edition customers willing to keep the Grafana Alerts enabled, use enableGrafanaAlerts feature flag:

company: thecompany
projectName: thedatahub
features:
  enableGrafanaAlerts: true

3.1.2 Git connectivity as part of commit process is deprecated

Replacing functionality is to use External API endpoints and custom git connectivity.

3.1.3 CUSTOM schedule is deprecated.

Replacing functionality is the Config Load Schedules package.

3.2 Other notable changes

• Default bastion host image of Centos Linux (end of life) is changed to AlmaLinux in Private Azure Edition.

• Workflow Orchestration (Apache Airflow) UI has changed a little bit in Grid/Graph views

  

• For triggering a Workflow and passing a parameter to run multiple of loads, the section Generated Configuration JSON is needed to be used/modified, and parameter load should be changed to loads.

  

3.3 Known Issues

In Designer in certain scenarios there might come an ugly error message filling whole UI screen. It is about JDBC exception executing SQL … ERROR: canceling statement due to user request and it comes when entering Data Model / Data Lineage / Workflows / Package Tree in certain scenarios. Refreshing the page in browser should make it disappear.