Release Notes 21.0.435
1. RELEASE INFO
Product | AGILE DATA ENGINE |
Release | 21.0.435 |
Release date | 2nd of July 2021 |
Release status | RELEASED |
Release type | MAJOR.MINOR.UPDATE.PATCH |
SaaS Availability | AVAILABLE | RESTRICTED (by request) |
AWS Availability | AVAILABLE | RESTRICTED (by request) |
Azure Availability | AVAILABLE | RESTRICTED (by request) |
Patch release scope | Not applicable | Design | Runtime | SaaS |
2. CONTENT OF THE RELEASE
2.1 Highlights
New External API for Run IDs
Major refactoring under the hood for Run ID Logic (renaming and logic to be based on Entity IDs and Load IDs instead of names).
New External API for Exporting Package JSON by Commit ID
New visualization modes (incl. user defined ones) and other UX improvements for visualization in Designer.
New Under Construction indicator as part of the package icon to indicate how many modifiers there are and on hover by whom and when.
New generic 1:1 SQL generation for MERGE, SWAP, OVERWRITE patterns based on Key and/or Datahash in Public Preview (support for Snowflake and Amazon Redshift). Used with new load options.
Satellite load performance improvement (not exists check with sub-query instead of current view). Requires re-generation of the loads.
View SQL can be templated and enhanced configurability introduced for entity types (now automatic current satellites can be defined for e.g. Business Satellites).
Private Link support for Amazon Redshift in SaaS Enterprise Editions.
Multi-DBMS support per Runtime in SaaS Enterprise Editions.
Manual demote support in Deployment Management.
PostgreSQL backup restore fix in Private Azure edition.
Support for Azure AKS 1.20.x in Private Azure edition.
See also the highlights of the major release: Release Notes 21.0
2.2 Security
Security fixes
ID | Description |
|---|---|
ADE‑XYZX | Some 3rd party software updates |
2.3 Infrastructure
Improvements and Fixes
ID | Description |
|---|---|
ADE‑3924 | Update AKS version to 1.20.x |
ADE‑3742 | Kubernetes internal load balancer cancels long lasting queries |
ADE‑3747 | aws cloudformation templates hitting size limit |
ADE‑3837 | AWS ADE Design env. Jenkins task started on ec2 that omits EFS mounts |
ADE‑3940 | Bastion create False causes problems in CF param bastionHostPublicIp |
ADE‑3951 | Ade backup restore fails because of non existent public schema |
ADE‑3956 | AKS cluster creation to 1.20 fails due to invalid addon |
2.4 Designer
New Features
ID | Description |
|---|---|
ADE‑3703 | New user defined visualization settings - Possible to change the way how lineage and entity graph is rendered. |
ADE‑3704 | Double-click should focus/unfocus entity in visualizations when in focused mode |
ADE‑3749 | Separate and store settings for Data model and Data lineage |
ADE‑3635 | Under construction icon for package level with information of users who have done uncommitted changes |
Improvements
ID | Description |
|---|---|
ADE‑3402 | Add Search for Physical Options, Keys, References, Permissions for App Search |
ADE‑3612 | Adding info about usage to app search |
ADE‑3718 | VIEW SQL cannot be saved as a template |
ADE‑3749 | Separate and store settings for Data model and Data lineage |
ADE‑1391 | Data Lineage draws arrows backwards |
Fixes
ID | Description |
|---|---|
ADE‑3723 | Focus entity is not in the url, if I share the url |
ADE‑3766 | When deleting entity which has outgoing loads, the error/info appears in opaque background |
ADE‑3715 | CSV export fails when two different keys are defined with same attributes |
2.5 Configurability
Improvements
ID | Description |
|---|---|
ADE‑3811 | Enable custom current view entity types |
2.6 Deployment Management
Improvements
ID | Description |
|---|---|
ADE‑3879 | Manual demote for a promoted package |
ADE‑3934 | Jenkins package job stage naming and help information |
ADE‑3827 | Adding HISTORIZED state to promotion state |
ADE‑3746 | Change deployment -> manager install call to be done as async |
Fixes
ID | Description |
|---|---|
ADE‑3720 | Commit Id should be stated instead of Revision in job description |
ADE‑3724 | Azure file listing is left empty |
ADE‑3738 | Deployment allowed to proceed for failing packages in case of future version succeeds |
ADE‑3740 | Jenkins package start waiting time not enough in migration |
ADE‑3771 | Freshly promoted package with manual deployment cleared out too early |
ADE‑3779 | Deployment-api external git branch is not configurable |
ADE‑3836 | Environment dependency order can not be changed |
ADE‑3884 | Deployment-api migration does not discover all package promotions |
ADE‑3886 | Investigate whether deployment api migration promotion could be skipped in case package jobs would be manually stopped |
ADE‑3955 | Index out of bounds error from deploy |
2.7 Loads and Workflows
New Features
ID | Description |
|---|---|
ADE‑3729 | Key and Key+Datahash based SQL generation for 1:1 mapped loads |
ADE‑3778 | Support for MERGE-logic |
ADE‑3801 | OPT_HISTORIZED_BY_DATAHASH_AS_INSERT |
Run ID Improvements
ID | Description |
|---|---|
ADE‑1615 | Run ID tech attr type and token name alignment |
Some of the Run ID logic related references are renamed. Workflow Orchestration (aka Dagger) is supporting the deprecated ones still, but be aware of below changes. | |
1. Technical attribute types SDT_STAGE_BATCH_ID and DV_RUN_ID are both renamed to RUN_ID. | |
2. Transformation type DV_RUN_ID is renamed to RUN_ID. | |
3. Variable <sourcebatchidattr> is renamed to <sourcerunidattr>. | |
4. Variable <targetbatchidattr> is renamed to <targetrunidattr>. | |
5. Variable <targetbatchid> is renamed to <targetrunid>. | |
6. Load option OPT_DV_RUN_IDS_PER_TRANSACTION is renamed to OPT_NUM_RUN_IDS_PER_TRANSACTION. | |
ADE‑3737 | Refactor Run ID logic and API to use Entity ID and Load ID instead of logical names |
1. Run ID logic is changed to be based on Entity IDs and Load IDs and the old Load name and Entity name based logic is not functioning anymore. | |
2. New Runtime partitioned metadata table is introduced P_RUN_ID (old one is P_RUNID). | |
3. New APIs internally in Process API are introduced based on the referred IDs. Old batch API is deprecated. | |
4. New APIs externally in External API are introduced for SaaS editions. |
Other Improvements and Fixes
ID | Description |
|---|---|
ADE‑3776 | OPT_NOT_EXISTS_WHERE support for generic 1:1 SQL generated loads for speeding source-target joining |
ADE‑3777 | Enable duplicate prevention to SIMPLE_INSERT |
ADE‑3853 | In CONFIG_LOAD_SCHEDULES setting empty value does not set the value as null |
ADE‑3844 | Satellite load performance improvement |
2.8 DBMS Support
Improvements and Fixes
ID | Description |
|---|---|
ADE‑3750 | Support for SUPER data type in Amazon Redshift |
ADE‑3835 | Update Google BigQuery JDBC driver |
ADE‑3882 | Amazon Redshift COPY prefers to use IAM_ROLE instead of CREDENTIALS |
ADE‑3744 | Variables references (tokens) not working in manifest load + Azure Synapse SQL + override_file_load |
2.9 Dashboards
Fixes
ID | Description |
|---|---|
ADE‑3550 | Logic error with Smoke results in Dashboard |
2.10 SaaS Editions
APIs
ID | Description |
|---|---|
ADE‑3089 | saas-external-api security improvements |
ADE‑3090 | saas-external-api security improvements |
ADE‑3832 | Publish commit details and content paths to external api |
ADE‑3512 | External API: Run ID APIs |
ADE‑3591 | Improvements to Notify API documentation (Swagger) |
Improvements
ID | Description |
|---|---|
ADE‑3728 | SaaS installer to support multi-dbms |
ADE‑3862 | SaaS installer to configure Snowflake warehouse variable/token |
ADE‑3864 | Package exclude/include support for SaaS Installer |
ADE‑3881 | JSON logs: dh-authenticator |
3. NOTIFICATIONS
3.1 Supported Upgrade Path
From 20.0.365 to 20.1
From 20.1 to 20.2.
From 20.2 to 21.0
Upgrading from earlier releases
First to 20.1.380.patch1 and then to 20.2 and then to 21.0.
E.g. from 19.3.334.patch4 to 20.0.354_patch7 and then directly 20.1.380.patch1 (or any latest of 20.1), then to 20.2.396.patch5 and finally to this release.
See earlier release notes also if upgrading from older releases.
3.2 Automatic migration mode need to be set for Deployment Management migration
Main configuration parameter deploymentApiMigrationMode should be set to value MINIMAL.
The migration will take place when Deployment API service starts and it will migrate the information of deployed and promoted and to-be-promoted packages into Deployment API's database.
Also, it will migrate the entity and config packages from build repository to new commit repository (based on the database and S3/Blob storage).
3.3 Automatic Metadata Repository Database snapshots in Private AWS edition
When ADE update detects release change or PostgreSQL version change in main configuration, database snapshots will be created from all ADE RDS PostgreSQL instances of that environment.
This happens by default as a post step in update and the retention time of 62 days is applied for these snapshots. This feature can be configured in main configuration if needed.
3.4 Private Azure edition: bastion hardening CentOS image has changed
If installing ADE to Private Azure or updating bastion to apply bastion hardening, the new Centos image to be used is defined and consult our Professional Services.
If trying to apply bastion hardening with earlier ADE release versions, it will fail. Corrected bastion hardening to this release.
This only concerns new Private Azure installations or if otherwise needed to re-install the bastion host with the hardening. Existing hardened bastions will work.
3.5 Private Azure edition: AKS cluster version upgrade
The AKS version will be upgraded to 1.18.10 by this release. Also AKS node pool version will be updated to the same as AKS version.
The update of the node pool version requires re-creation of the AKS cluster.
3.6 Metadata Repository Database's PostgreSQL version upgrade
If not yet done with earlier release, in AWS edition the PostgreSQL version will be upgraded from current 9.6 to 11. This took about 25 minutes per environment in our test environment. Dependent on the database sizes.
In Private Azure edition, the upgrade is not done by default. It must be done by you manually. The PostgreSQL upgrade from 9.x to 11 should already be done with earlier release 20.2.
3.7 Private Azure edition: PostgresSQL version check
ADE update in Private Azure will now check that the PostgresSQL version is 11 (The version provided by ADE release currently) and if not, an exception is thrown and update can't be continued before upgraded to 11.
3.8 packageversion token value has changed and it will get longer values than earlier
Originally <packageversion> token has gotten the value from Jenkins build number ranging usually from 1 to 10000.
From 21.0 onwards the token will get the value from Deployment API's commit id, which is global sequence per installation and it starts from value 100000 for new commits and from value 500000 for migrated packages. So when using <packageversion> token with strictly limited varchar/char lengths in a column, this can cause failure or truncated value at least.
3.9 Related to Grafana upgrade, old alive session might cause problem in login
NOTE: If having problems logging in to Grafana, it will need to clear old Grafana sessions from the web browser. Open dev tools and sessions and remove Grafana related.
3.10 Related to Grafana upgrade, AWS Cloudwatch datasource can fail in Private AWS editions
NOTE: If having AWS Cloudwatch datasource, it might switch provider to "credential". This will fail since there is no any credential files in Grafana service. The provider should be "AWS SDK Default", it will use the granted AWS role for Cloudwatch.
3.11 There is new Run ID API and the old batch API is deprecated
There is new implementation of the Run ID API. The old one is deprecated and will not function (only getting new batchId works). Workflow Orchestration (aka. Dagger) will use the new one.
There is no user action needed unless old APIs are used by some custom implementation. The change is that the new one uses Entity IDs and Load IDs instead of Entity and Load names.
The Runtime internal service Process API serves the API as before and makes migration of Run ID history into new table structure P_RUN_ID (old is P_RUNID).
3.12 Additional options to use with update scripts
For speeding up the update process:
packages-only: This will download ADE package and sync docker images to local machine and to target repository, but not run the installation.
Consider loading docker images to target ADE environment by this option beforehand. This update operation will not change anything in that environment except for adding new images to repositories.
skip-packages: Will skip the docker image downloading and syncing to target repo. Useful if needed to iterate ADE configuration changes etc.
E.g: ./update_ade_design.sh the-main-conf.yaml DESIGN skip-packages