Release Notes 21.0.420
1. RELEASE INFO
Product | AGILE DATA ENGINE |
Release | 21.0.420 |
Release date | 13th of April 2021 |
Release status | RELEASED |
Release type | MAJOR.MINOR.UPDATE.PATCH |
SaaS Availability | AVAILABLE | RESTRICTED (by request) |
AWS Availability | AVAILABLE | RESTRICTED (by request) |
Azure Availability | AVAILABLE | RESTRICTED (by request) |
Patch release scope | Not applicable | Design | Runtime | SaaS |
2. CONTENT OF THE RELEASE
2.1 Highlights
Few fixes to new Deployment API introduced in 21.0
To start workflow/DAG run with parameters: load name, list of load names, package name
Dashboard software Grafana is upgraded to 7
3rd party software updates with security fixes
See also the highlights of the major release: Release Notes 21.0
2.2 Security
Security fixes
ID | Description |
|---|---|
ADE‑3274 | Grafana upgrade to 7.x. See Notifications: 3.10 |
ADE‑3690 | Designer 3rd party software updates |
2.3 Infrastructure
Improvements
ID | Description |
|---|---|
ADE‑3717 | Update AKS version to 1.18.14 |
Fixes
ID | Description |
|---|---|
ADE‑3714 | Infrastructure asks for old components |
2.4 Deployment Management
Fixes
ID | Description |
|---|---|
ADE‑3531 | Jenkins shows times in different time zones |
ADE‑3695 | Package installation order resolution performance problem |
ADE‑3708 | Deployment-api not stopping package installation in case of physical metadata generation problem |
ADE‑3716 | Jenkins package polling fails to continue if authentication error occurs |
ADE‑3709 | Deployment job execution should not stop to failure in case of nothing to deploy |
ADE‑3712 | Resolve system user also in Manager |
2.5 Loads and Workflows
New Features
ID | Description |
|---|---|
ADE‑3667 | To start workflow/DAG run with parameters: load name, list of load names, package name |
Parameter type | Example | Description |
|---|---|---|
on_error_pause_dag boolean | {"on_error_pause_dag": true} | If task fails workflow/DAG will get paused. Defaults to schedule config. |
load string | {"load": "load_name_to_run"} | Only load with given name is run. If value is empty then it is not affecting. |
loads string array | {"loads": ["load_name_to_run", "load_name_to_run_2"]} | Only loads with given names are run. If value is empty list then it is not affecting. |
package string | {"package": "package_of_loads_to_run"} | Only loads with given package name are run. If value is empty then it is not affecting. |
packages string array | {"packages": ["package_of_loads_to_run", "package_of_loads_to_run_2"]} | Only loads with given package names are run. If value is empty list then it is not affecting. |
trigger_workflows string array | {"trigger_workflows":["WORKFLOW_TO_TRIGGER"]} | Only listed workflow triggers are run. Notice you can't trigger workflow which is not configured to be triggered in schedule configuration. This can be only used to prevent existing triggers to trigger. |
2.6 DBMS Support
Fixes
ID | Description |
|---|---|
ADE‑3207 | Materialized view recovery after modification failure fails |
2.7 SaaS Editions
APIs
ID | Description |
|---|---|
ADE‑3713 | Notify API: Change Dagger term to Workflow Orchestration |
3. NOTIFICATIONS
3.1 Supported Upgrade Path
From 20.0.365 to 20.1
From 20.1 to 20.2.
From 20.2 to 21.0
Upgrading from earlier releases
First to 20.1.380.patch1 and then to 20.2 and then to 21.0.
E.g. from 19.3.334.patch4 to 20.0.354_patch7 and then directly 20.1.380.patch1 (or any latest of 20.1), then to 20.2.396.patch5 and finally to 21.0.420.
See earlier release notes also if upgrading from older releases.
3.2 Automatic migration mode need to be set for Deployment Management migration
Main configuration parameter deploymentApiMigrationMode should be set to value MINIMAL.
The migration will take place when Deployment API service starts and it will migrate the information of deployed and promoted and to-be-promoted packages into Deployment API's database.
Also, it will migrate the entity and config packages from build repository to new commit repository (based on the database and S3/Blob storage).
3.3 Automatic Metadata Repository Database snapshots in Private AWS edition
When ADE update detects release change or PostgreSQL version change in main configuration, database snapshots will be created from all ADE RDS PostgreSQL instances of that environment.
This happens by default as a post step in update and the retention time of 62 days is applied for these snapshots. This feature can be configured in main configuration if needed.
3.4 Obsolete NAT gateway removed if exists in Private AWS edition
In ADE created network having multiple public subnets, there are also multiple Nat GW's. Those are reduced to one.
This also results reducing multiple outbound public IP's to one. So would be good to remove excess public IP's from any whitelistings.
Outbound public IP's will be also logged out in runtime ade update post step
e.g: "infrarun.infrarun.rel.post_steps - INFO - Outbound public IPs: 52.30.120.126"
3.5 Private Azure edition: bastion hardening CentOS image has changed
If installing ADE to Private Azure or updating bastion to apply bastion hardening, the new Centos image to be used is defined and consult our Professional Services.
If trying to apply bastion hardening with earlier ADE release versions, it will fail. Corrected bastion hardening to this release.
This only concerns new Private Azure installations or if otherwise needed to re-install the bastion host with the hardening. Existing hardened bastions will work.
3.6 Private Azure edition: AKS cluster version upgrade
The AKS version will be upgraded to 1.18.10 by this release. Also AKS node pool version will be updated to the same as AKS version.
The update of the node pool version requires re-creation of the AKS cluster.
3.7 Metadata Repository Database's PostgreSQL version upgrade
If not yet done with earlier release, in AWS edition the PostgreSQL version will be upgraded from current 9.6 to 11. This took about 25 minutes per environment in our test environment. Dependent on the database sizes.
In Private Azure edition, the upgrade is not done by default. It must be done by you manually. The PostgreSQL upgrade from 9.x to 11 should already be done with earlier release 20.2.
3.8 Private Azure edition: PostgresSQL version check
ADE update in Private Azure will now check that the PostgresSQL version is 11 (The version provided by ADE release currently) and if not, an exception is thrown and update can't be continued before upgraded to 11.
3.9 packageversion token value has changed and it will get longer values than earlier
Originally <packageversion> token has gotten the value from Jenkins build number ranging usually from 1 to 10000.
From 21.0 onwards the token will get the value from Deployment API's commit id, which is global sequence per installation and it starts from value 100000 for new commits and from value 500000 for migrated packages. So when using <packageversion> token with strictly limited varchar/char lengths in a column, this can cause failure or truncated value at least.
3.10 Related to Grafana upgrade, old alive session might cause problem in login
NOTE: If having problems logging in to Grafana, it will need to clear old Grafana sessions from the web browser. Open dev tools and sessions and remove Grafana related.
3.11 Additional options to use with update scripts
For speeding up the update process:
packages-only: This will download ADE package and sync docker images to local machine and to target repository, but not run the installation.
Consider loading docker images to target ADE environment by this option beforehand. This update operation will not change anything in that environment except for adding new images to repositories.
skip-packages: Will skip the docker image downloading and syncing to target repo. Useful if needed to iterate ADE configuration changes etc.
E.g: ./update_ade_design.sh the-main-conf.yaml DESIGN skip-packages