Release Notes 20.2
1. RELEASE INFO
Product | AGILE DATA ENGINE |
Release | 20.2 (20.2.394) |
Release date | 16th of December 2020 |
Release status | RELEASED |
Release type | MAJOR.MINOR.UPDATE.PATCH |
SaaS Availability | AVAILABLE | RESTRICTED (by request) |
AWS Availability | AVAILABLE | RESTRICTED (by request) |
Azure Availability | AVAILABLE | RESTRICTED (by request) |
Patch release scope | Not Applicable | Design | Runtime | SaaS |
2. CONTENT OF THE RELEASE
2.1 Highlights
Metadata Repository DB's PostgreSQL upgrade to version 11
In Private AWS and SaaS Editions as part of ADE upgrade.
In Private Azure manually after ADE upgrade.
Tooling and manual procedure provided for Private Azure due Azure limitations. In Azure the current 9.5 will deprecate in Feb 2021.
Read the Notifications and the Manual upgrade instructions in these release notes.
New Deployment API v1 replaces Builder API and Version Control API.
In the next release Deployment API v2 will replace lots of logic implemented currently into Jenkins.
Improvements to Private Azure Infrastructure
PgBouncer as its own versioned component
Tag support
AKS version update
Etc.
Lots of improvements in Designer
E.g. new commit message format validation framework with Config Validations.
SaaS related
New SaaS Designer Only Option
Automatic SaaS Installer v1
Cloud foundation improved
2.2 Security
Security fixes
ID | Description |
|---|---|
ADE‑3068 | Upgrade Spring Boot and cleanup dependency handling of Process API |
ADE‑3069 | Upgrade Spring Boot and cleanup dependency handling of Notify API |
2.3 Infrastructure
Improvements
ID | Description |
|---|---|
ADE‑2168 | PostgreSQL db version upgrades in SaaS, AWS and Azure editions |
ADE‑3035 | New dh-azure-pgbouncer component to ensure tested version in a release |
ADE‑3009 | Replace builder-api and versioncontrol-api with new deployment-api |
ADE‑3065 | Update AKS version in Azure edition |
ADE‑3076 | Tag support in Terraform resources in Azure edition |
ADE‑3094 | Jenkins viewable build amount in stage view configurable |
ADE‑3083 | Possible to skip TF apply in ade_update script run |
ADE‑3108 | Azure add Microsoft.Storage private endpoint |
ADE‑3123 | AWS create manual snapshot of RDS PostgreSQL in ADE update |
ADE‑2969 | Idea of instructing tee command usage during updates |
Fixes
ID | Description |
|---|---|
ADE‑3075 | Remove obsolete NatGw from AWS network |
2.4 Designer
New Features
ID | Description |
|---|---|
ADE‑3024 | New Config Package: CONFIG_VALIDATIONS and first implementation of commit message validation. |
ADE‑2914 | Config package dependency list in Designer |
ADE‑2989 | Add section of Triggering Schedules (non-editable) to Schedule details view |
ADE‑3060 | Template support for default attribute's name |
ADE‑2960 | Support searching schedules with app search |
User Experience Improvements
ID | Description |
|---|---|
ADE‑2939 | Creating a view with GENERIC entity type overwrites the schema when name is changed |
ADE‑2950 | Unnecessary scroll bars in load step code view |
ADE‑3049 | Reference create/edit: minor issue when switching parent entities |
ADE‑3059 | User defined schema is overwritten with default, when entity name is touched |
ADE‑3098 | Using copy by ref should be available for hub dv_id references in links also |
ADE‑2893 | References: Schema aware list of parent entities |
ADE‑2991 | When deleting schedule, there should be validation of existing loads referring to the schedule |
ADE‑3084 | When new entity is created in Entity listing, the view should change to Entity summary |
ADE‑3145 | Longer Description field for Load Transformations |
Fixes
ID | Description |
|---|---|
ADE‑3056 | Order problem in SQL Export for MS SQL SQL in Designer |
ADE‑3140 | Change CREATE PROCEDURE into CREATE OR ALTER PROCEDURE in SQL Export for MS SQL |
ADE‑3080 | CREATE PROCEDURE is generated for each load step alone and even with the same name in SQL Export e.g. in case of Status Satellites |
ADE‑3034 | Rename 'Schedule loads' to 'Scheduled loads' |
ADE‑3079 | Entity export in CSV does not include attribute_is_soft_deleted field |
ADE‑3141 | Commit history is not returned for deleted package |
ADE‑3161 | Entity deep clone sets cloned loads as disabled |
ADE‑3171 | Entity/Satellite deletion hangs and won't go through with couple of hundred attributes |
2.5 Configurability
New Features
ID | Description |
|---|---|
ADE‑3024 | New Config Package: CONFIG_VALIDATIONS and first implementation of commit message validation |
2.6 Deployment Management
New Features
ID | Description |
|---|---|
ADE‑3009 | Replace builder-api and versioncontrol-api components with new deployment-api |
ADE‑3094 | Jenkins viewable build amount in stage view configurable |
Fixes
ID | Description |
|---|---|
ADE‑3077 | Cleared masking policy or collation will fail in entity recreation of the entity |
ADE‑3078 | Entity re-create loses table definitions e.g. columnstore definition |
ADE‑3170 | Rollback problem with Snowflake in entity change in certain case |
2.7 Loads and Workflows
Improvements
ID | Description |
|---|---|
ADE‑2942 | Add Batch ID and executed SQL query to failed smoke test result messages |
ADE‑3073 | Alias name for DV loads source entities |
ADE‑2992 | Prevention of SELECTs from load steps to ensure customer data won't be transferred to ADE runtime in SaaS editions |
2.8 DBMS Support
Fixes
ID | Description |
|---|---|
ADE‑3077 | Cleared masking policy or collation will fail in entity recreation of the entity |
ADE‑3078 | Entity re-create loses table definitions e.g. columnstore definition |
ADE‑3170 | Rollback problem with Snowflake in entity change in certain case |
2.9 SaaS Editions
APIs
Development of new External API implementation has progressed and limited preview status is expected in the 21.0 release for Enterprise Edition.
Improvements
Lots of improvements in SaaS foundation
Lots of development done for upcoming External API
ID | Description |
|---|---|
ADE‑2992 | Prevention of SELECTs from load steps to ensure customer data won't be transferred to ADE runtime |
ADE‑3081 | Designer Only SaaS Option ie. Possiblility to disable Deployment Management from AWS installation |
3. NOTIFICATIONS
3.1 Supported Upgrade Path
From 20.0.365 to 20.1
From 20.1 to 20.2.
Upgrading from earlier releases
First to 20.1.380 and then directly to 20.2.
E.g. from 19.3.334.patch4 to 20.0.354.patch6 and then directly 20.1.380 (Or any latest of 20.1) and then to 20.2
See earlier release notes also if upgrading from older releases.
3.2 Automatic Metadata Repository Database snapshots in Private AWS edition
When ADE update detects release change or PostgreSQL version change in main configuration, database snapshots will be created from all ADE RDS PostgreSQL instances of that environment.
This happens by default as a post step in update and the retention time of 62 days is applied for these snapshots. This feature can be configured in main configuration if needed.
3.3 Metadata Repository Database's PostgreSQL version upgrade
In AWS edition the PostgreSQL version will be upgraded from current 9.6 to 11. This will take about 25 minutes per environment.
In Azure edition, the upgrade is not done by default. It must be done by you manually after release upgrade at some point before the next release update.
3.4 Obsolete NAT gateway removed if exists in Private AWS edition
In ADE created network having multiple public subnets, there are also multiple Nat GW's. Those are reduced to one.
This also results reducing multiple outbound public IP's to one. So would be good to remove excess public IP's from any whitelistings.
Outbound public IP's will be also logged out in runtime ade update post step
e.g: "infrarun.infrarun.rel.post_steps - INFO - Outbound public IPs: 52.30.120.126"
3.5 Additional options to use with update scripts
For speeding up the update process:
packages-only: This will download ADE package and sync docker images to local machine and to target repository, but not run the installation.
Consider loading docker images to target ADE environment by this option beforehand. This update operation will not change anything in that environment except for adding new images to repo's.
skip-packages: Will skip the docker image downloading and syncing to target repo. Useful if needed to iterate ADE configuration changes etc.
E.g: ./update_ade_design.sh the-main-conf.yaml DESIGN skip-packages