1. RELEASE INFO
|
|
|
|---|---|
|
Product |
AGILE DATA ENGINE |
|
Release |
21.0 |
|
Release date |
29th of March 2021 |
|
Release status |
RELEASED |
|
Release type |
MAJOR.MINOR.UPDATE.PATCH |
|
SaaS Availability |
AVAILABLE | RESTRICTED (by request) |
|
AWS Availability |
AVAILABLE | RESTRICTED (by request) |
|
Azure Availability |
AVAILABLE | RESTRICTED (by request) |
|
Patch release scope |
Not applicable | Design | Runtime | SaaS |
2. CONTENT OF THE RELEASE
2.1 Highlights
New Deployment API v2
-
Deployment Management has gone through major refactoring into API based implementation with many small improvements.
-
In earlier releases the new Deployment API has combined the functionality of Version Control API and Builder API already.
-
In this release whole deployment management logic has been moved into Deployment API and new Deployment Management database.
-
There is new concept of Commit Repository and local git has been removed fully and external git is supported as one direction integration (push) only.
-
Jenkins is used fully as UI frontend for the Deployment API and there are several improvements in Jenkins UI/UX implemented.
-
There is automatic migration of old build repository (S3/Blob) and deployment and promotion statuses per runtime environment into commit repository and deployment management database.
-
New Deployment API will open up new possibilities for API based integrations and for new Deployment Management UI in future.
-
In this release there is the first implementation of External API (in SaaS Enterprise Edition) to retrieve commit message list between promoted and deployed package versions per environment.
New External API
-
Multi-tenant serverless external API service in SaaS Enterprise Edition (in limited preview).
-
This new service is the framework to have API implementations for the metadata and for different integration needs.
-
Allowed IP address restriction is implemented on API key basis.
-
There are two new APIs implemented: 1) Commit message list between promoted and deployed package versions per environment metadata API and 2) Workflow/DAG start and status control APIs.
New graphical modelling experience in preview
-
There is possibility to model (create/edit/delete entities and references) in the graphical data model view
New support for Snowflake SHAREs and Amazon Redshift's AMI_ROLEs
-
Defining grantee types for grantees is supported with CONFIG_ENTITY_PERMISSIONS package
-
Designer's Entity Permissions tab has new sections per grantee type
-
All the current grantees are migrated to grantee type ROLE (synonym for Amazon Redshift's GROUP)
Notify API Swagger UX improved
-
Restructured UI and inline documentation added
Third party software updates
-
JBoss Keycloak is updated
-
Swagger and Spring boot updated
-
AKS version updated
-
And following update releases will include upgrade of Grafana 7.0 and there will be also upgrade of Apache Airflow 2.0.
PostgreSQL version 11 supported only without exceptions
-
In Private Azure edition the PostgreSQL version upgrade is required before upgrading to Agile Data Engine 21.0 version
-
In Private AWS edition the PostgreSQL version upgrade is not forced, but it is expected, and should have happened automatically in 20.2 installation already.
2.2 Security
Security fixes
|
ID |
Description |
|---|---|
|
ADE‑XYZ |
Several security fixes and 3rd party component updates |
|
... |
... |
|
ADE‑3177 |
Update Jenkins version |
|
ADE‑3272 |
Update Grafana version |
|
ADE‑3418 |
Snowflake jdbc driver update |
|
ADE‑3552 |
BigQuery JDBC driver update |
2.3 Infrastructure
Improvements
|
ID |
Description |
|---|---|
|
ADE‑3459 |
Reconfigure ecs cluster to utilize t3a.small by default and configure reserverd memory better to avoid instance level memory problems |
|
ADE‑3515 |
Upgrade AKS version to 1.18 |
|
ADE‑3124 |
Azure ADE release should check that postgres version is upgraded |
|
ADE‑3527 |
Dagger: install upgrade check scripts to current image |
Fixes
|
ID |
Description |
|---|---|
|
ADE‑3373 |
Infra maintenance dagger stop wait expires too soon |
|
ADE‑3268 |
ADE bastion update users script should look main json from main conf location |
|
ADE‑3286 |
infra-maintenance fails dagger_start if dag removed from Dagger |
|
ADE‑3297 |
Format LocalDateTimes as ISO date time format in REST APIs |
|
ADE‑3458 |
AWS Lambda will no longer support the botocore.requests library in Python runtimes. If you are using the cfn-response module for a Lambda-backed custom resource... |
|
ADE‑3634 |
Azure manager workflow generation fails to overwrite |
|
ADE‑3639 |
Azure bastion hardening fails to install |
2.4 Designer
New Features
|
ID |
Description |
|---|---|
|
ADE‑2986 |
Entity and Reference creation in Data Model |
|
ADE‑3160 |
DW Zone filters in Data Model and Data Lineage |
|
ADE‑3402 |
Add Search API for Physical Options, Keys, References, Permissions for App Search |
|
ADE‑3403 |
New token: <target_entity_logical_name> to be used in the loads |
User Experience Improvements
|
ID |
Description |
|---|---|
|
ADE‑3611 |
App search improvements to support workflow using multiple tabs |
|
ADE‑3413 |
App search: Separate section for search filters |
|
ADE‑3614 |
List references in the order of child attribute position |
|
ADE‑3620 |
Upgrade of vis.js library for Visualizations |
|
ADE‑3631 |
Replace entity mappings' arrow-balloon-arrow with an arrow in Data Lineage |
|
ADE‑3632 |
Place cursor by default to search box after Create entity mapping clicked |
|
ADE‑3398 |
When deleted entity is changed and saved, the is_deleted is changed to false without confirmation |
|
ADE‑3422 |
SaaS: back- & search-icons too dark when hovering |
|
ADE‑3518 |
Load enabled/disabled info is not user friendly in loads' listing |
|
ADE‑3553 |
Pressing enter in app search not working properly when app search is clicked open |
|
ADE‑3569 |
dbms product colour vanish to background |
|
ADE‑3159 |
Check and align the form field descriptions stylings |
|
ADE‑3239 |
When changing VIEW to TABLE (physical type) the TRANSFORM_SQL_VIEW type of a load should be changed to TRANSFORM_PERSIST |
|
ADE‑3336 |
Align boldness for entity and load types |
Fixes
|
ID |
Description |
|---|---|
|
ADE‑3385 |
GATEKEEPER step always as the first one in SQL Export |
|
ADE‑3389 |
Cannot reorder attribute positions for existing entities & attributes with Entity import |
|
ADE‑3454 |
Importing multiple entities with same primary key name creates one primary key with multiple attributes |
|
ADE‑3517 |
A load deletion does not show any recent changes for a package |
|
ADE‑3590 |
Map all mapped does not function in all cases |
|
ADE‑3298 |
Load create/edit forms: change 'scheduling' term to 'schedule' |
|
ADE‑3305 |
Incoming load creation form: cron expression field could be smaller |
|
ADE‑3295 |
Fix commit sizes to not be optional (like model is already defined) |
|
ADE‑3296 |
1664+ entity package import fails in metarepository-api |
|
ADE‑3299 |
Load edit forms: align form field width sizes |
|
ADE‑3300 |
Load edit form (in detailed page): no schedule fields are shown if entity is 'VIEW'' |
|
ADE‑3412 |
Select inputs: align field height and margin between error message |
|
ADE‑2750 |
Setting physical option and double-clicking the add button, it returns a duplicate error in Designer |
|
ADE‑3041 |
Edit reference: failure when changing the parent entity |
|
ADE‑3043 |
Edit reference: don't show the [x] button next to attributes pair if it's the only available pair |
|
ADE‑3044 |
Edit load info form: case when cron expression doesn't get updated on schedule selection change |
|
ADE‑3329 |
Clicking on 'back' button while being in model view results in a failure |
|
ADE‑3410 |
Model view search sub panel is not updating the entity list in some cases |
|
ADE‑3460 |
Each click in data lineage renders the graph anew |
2.5 Configurability
New Features
|
ID |
Description |
|---|---|
|
ADE‑3416 |
Addition of grantee type to CONFIG_ENTITY_PERMISSIONS to enable permissions to Snowflake SHAREs |
2.6 Deployment Management
New Features
|
ID |
Description |
|---|---|
|
ADE‑3005 |
New Deployment API |
|
ADE‑3513 |
Jenkins job stylings based on job status |
|
ADE‑3324 |
dh-metarepository-api: Out of memory from app (increase memory in Private AWS) |
Improvements
|
ID |
Description |
|---|---|
|
ADE‑3317 |
Investigate Jenkins lengthy pipeline processing for package deployments having NO_CHANGES |
|
ADE‑3320 |
Improve update workflows in case of S3 to avoid unnecessary file transfers |
|
ADE‑3321 |
Skip deployment run if newer one exists (special optimization) |
|
ADE‑3382 |
Align names, titles and favicons with Deployment Management and Workflow Orchestration |
|
ADE‑3417 |
Package import speed improvements |
|
ADE‑3463 |
Disable system pre load job from Jenkins |
|
ADE‑3464 |
Refactor: Gson to Jackson library and remove Metadata API DTO dependency |
|
ADE‑3465 |
Change the "Time waiting in queue" in Jenkins jobs to zero seconds |
|
ADE‑3521 |
Provide installation information also to the package job logs |
|
ADE‑3529 |
Make package installation message more clear |
|
ADE‑3532 |
Make package name in top breadcrumb bolder |
|
ADE‑3533 |
Make package statuses bolder |
|
ADE‑3541 |
Package deployment environment selection should indicate the action to be promotion |
|
ADE‑3542 |
Change package build commitMessage to buildInfo and make it extend the actional commit message |
|
ADE‑3545 |
Add visualization for deployment management |
|
ADE‑3562 |
Update new favicon |
Fixes
|
ID |
Description |
|---|---|
|
ADE‑3322 |
Looks like metarepo api converter class used in import and export of packages may have request specific state and it still used as singleton |
|
ADE‑3407 |
Manager StackOverflow with large pkg |
|
ADE‑3462 |
Enable Hibernate's query plan cache parameters to avoid some of potential excessive memory usage problems |
|
ADE‑3638 |
Double encoding with rest template |
|
ADE‑2925 |
Showing logs in overlay views in jenkins are not working with theme colors at all |
|
ADE‑3307 |
Package installation failure not descriptive in Jenkins |
|
ADE‑3318 |
Use JWT originated author for commit instead of trusting client provided info in dto |
|
ADE‑3615 |
Deploy error does not show the related SQL statement |
2.7 Loads and Workflows
Improvements
|
ID |
Description |
|---|---|
|
ADE‑3392 |
Cannot refer to relative paths with manifest and MS DATA SOURCES ie. Storage Integration |
|
ADE‑3382 |
Align names, titles and favicons with Deployment Management and Workflow Orchestration |
|
ADE‑3403 |
New token: <target_entity_logical_name> to be used in the loads |
|
ADE‑3559 |
Notify API and Executor to support Parquet, Avro and ORC files in addition |
2.8 DBMS Support
New Features
|
ID |
Description |
|---|---|
|
ADE‑3416 |
Addition of grantee type to CONFIG_ENTITY_PERMISSIONS to enable permissions to Snowflake SHAREs |
|
ADE‑3187 |
Force SSL for Redshift connections in Manager and Executor |
Fixes
|
ID |
Description |
|---|---|
|
ADE‑3418 |
Snowflake jdbc driver should be updated before 30th April 2021 |
|
ADE‑3552 |
BigQuery JDBC driver update |
2.9 SaaS Related
APIs
|
ID |
Description |
|---|---|
|
ADE‑2139 |
External API Service Component |
|
ADE‑3609 |
Notify API: Swagger theme improvements for markdown |
|
ADE‑3657 |
Notify creation fails with aws s3 client error |
|
ADE‑3559 |
Notify API and Executor to support Parquet, Avro and ORC files in addition |
Improvements
|
ID |
Description |
|---|---|
|
ADE‑3198 |
saas-installer to add elb access logging |
|
ADE‑3337 |
Grafana user role to editor role |
|
ADE‑3421 |
Theme update for Grafana 7.x |
|
ADE‑3660 |
SaaS installer to support deployment-api migration mode |
Fixes
|
ID |
Description |
|---|---|
|
ADE‑3658 |
Modified user not shown in Designer |