Release Notes 21.0
1. RELEASE INFO
Product | AGILE DATA ENGINE |
Release | 21.0 |
Release date | 29th of March 2021 |
Release status | RELEASED |
Release type | MAJOR.MINOR.UPDATE.PATCH |
SaaS Availability | AVAILABLE | RESTRICTED (by request) |
AWS Availability | AVAILABLE | RESTRICTED (by request) |
Azure Availability | AVAILABLE | RESTRICTED (by request) |
Patch release scope | Not applicable | Design | Runtime | SaaS |
2. CONTENT OF THE RELEASE
2.1 Highlights
New Deployment API v2
Deployment Management has gone through major refactoring into API based implementation with many small improvements.
In earlier releases the new Deployment API has combined the functionality of Version Control API and Builder API already.
In this release whole deployment management logic has been moved into Deployment API and new Deployment Management database.
There is new concept of Commit Repository and local git has been removed fully and external git is supported as one direction integration (push) only.
Jenkins is used fully as UI frontend for the Deployment API and there are several improvements in Jenkins UI/UX implemented.
There is automatic migration of old build repository (S3/Blob) and deployment and promotion statuses per runtime environment into commit repository and deployment management database.
New Deployment API will open up new possibilities for API based integrations and for new Deployment Management UI in future.
In this release there is the first implementation of External API (in SaaS Enterprise Edition) to retrieve commit message list between promoted and deployed package versions per environment.
New External API
Multi-tenant serverless external API service in SaaS Enterprise Edition (in limited preview).
This new service is the framework to have API implementations for the metadata and for different integration needs.
Allowed IP address restriction is implemented on API key basis.
There are two new APIs implemented: 1) Commit message list between promoted and deployed package versions per environment metadata API and 2) Workflow/DAG start and status control APIs.
New graphical modelling experience in preview
There is possibility to model (create/edit/delete entities and references) in the graphical data model view
New support for Snowflake SHAREs and Amazon Redshift's AMI_ROLEs
Defining grantee types for grantees is supported with CONFIG_ENTITY_PERMISSIONS package
Designer's Entity Permissions tab has new sections per grantee type
All the current grantees are migrated to grantee type ROLE (synonym for Amazon Redshift's GROUP)
Notify API Swagger UX improved
Restructured UI and inline documentation added
Third party software updates
JBoss Keycloak is updated
Swagger and Spring boot updated
AKS version updated
And following update releases will include upgrade of Grafana 7.0 and there will be also upgrade of Apache Airflow 2.0.
PostgreSQL version 11 supported only without exceptions
In Private Azure edition the PostgreSQL version upgrade is required before upgrading to Agile Data Engine 21.0 version
In Private AWS edition the PostgreSQL version upgrade is not forced, but it is expected, and should have happened automatically in 20.2 installation already.
2.2 Security
Security fixes
ID | Description |
|---|---|
ADE‑XYZ | Several security fixes and 3rd party component updates |
... | ... |
ADE‑3177 | Update Jenkins version |
ADE‑3272 | Update Grafana version |
ADE‑3418 | Snowflake jdbc driver update |
ADE‑3552 | BigQuery JDBC driver update |
2.3 Infrastructure
Improvements
ID | Description |
|---|---|
ADE‑3459 | Reconfigure ecs cluster to utilize t3a.small by default and configure reserverd memory better to avoid instance level memory problems |
ADE‑3515 | Upgrade AKS version to 1.18 |
ADE‑3124 | Azure ADE release should check that postgres version is upgraded |
ADE‑3527 | Dagger: install upgrade check scripts to current image |
Fixes
ID | Description |
|---|---|
ADE‑3373 | Infra maintenance dagger stop wait expires too soon |
ADE‑3268 | ADE bastion update users script should look main json from main conf location |
ADE‑3286 | infra-maintenance fails dagger_start if dag removed from Dagger |
ADE‑3297 | Format LocalDateTimes as ISO date time format in REST APIs |
ADE‑3458 | AWS Lambda will no longer support the botocore.requests library in Python runtimes. If you are using the cfn-response module for a Lambda-backed custom resource... |
ADE‑3634 | Azure manager workflow generation fails to overwrite |
ADE‑3639 | Azure bastion hardening fails to install |
2.4 Designer
New Features
ID | Description |
|---|---|
ADE‑2986 | Entity and Reference creation in Data Model |
ADE‑3160 | DW Zone filters in Data Model and Data Lineage |
ADE‑3402 | Add Search API for Physical Options, Keys, References, Permissions for App Search |
ADE‑3403 | New token: <target_entity_logical_name> to be used in the loads |
User Experience Improvements
ID | Description |
|---|---|
ADE‑3611 | App search improvements to support workflow using multiple tabs |
ADE‑3413 | App search: Separate section for search filters |
ADE‑3614 | List references in the order of child attribute position |
ADE‑3620 | Upgrade of vis.js library for Visualizations |
ADE‑3631 | Replace entity mappings' arrow-balloon-arrow with an arrow in Data Lineage |
ADE‑3632 | Place cursor by default to search box after Create entity mapping clicked |
ADE‑3398 | When deleted entity is changed and saved, the is_deleted is changed to false without confirmation |
ADE‑3422 | SaaS: back- & search-icons too dark when hovering |
ADE‑3518 | Load enabled/disabled info is not user friendly in loads' listing |
ADE‑3553 | Pressing enter in app search not working properly when app search is clicked open |
ADE‑3569 | dbms product colour vanish to background |
ADE‑3159 | Check and align the form field descriptions stylings |
ADE‑3239 | When changing VIEW to TABLE (physical type) the TRANSFORM_SQL_VIEW type of a load should be changed to TRANSFORM_PERSIST |
ADE‑3336 | Align boldness for entity and load types |
Fixes
ID | Description |
|---|---|
ADE‑3385 | GATEKEEPER step always as the first one in SQL Export |
ADE‑3389 | Cannot reorder attribute positions for existing entities & attributes with Entity import |
ADE‑3454 | Importing multiple entities with same primary key name creates one primary key with multiple attributes |
ADE‑3517 | A load deletion does not show any recent changes for a package |
ADE‑3590 | Map all mapped does not function in all cases |
ADE‑3298 | Load create/edit forms: change 'scheduling' term to 'schedule' |
ADE‑3305 | Incoming load creation form: cron expression field could be smaller |
ADE‑3295 | Fix commit sizes to not be optional (like model is already defined) |
ADE‑3296 | 1664+ entity package import fails in metarepository-api |
ADE‑3299 | Load edit forms: align form field width sizes |
ADE‑3300 | Load edit form (in detailed page): no schedule fields are shown if entity is 'VIEW'' |
ADE‑3412 | Select inputs: align field height and margin between error message |
ADE‑2750 | Setting physical option and double-clicking the add button, it returns a duplicate error in Designer |
ADE‑3041 | Edit reference: failure when changing the parent entity |
ADE‑3043 | Edit reference: don't show the [x] button next to attributes pair if it's the only available pair |
ADE‑3044 | Edit load info form: case when cron expression doesn't get updated on schedule selection change |
ADE‑3329 | Clicking on 'back' button while being in model view results in a failure |
ADE‑3410 | Model view search sub panel is not updating the entity list in some cases |
ADE‑3460 | Each click in data lineage renders the graph anew |
2.5 Configurability
New Features
ID | Description |
|---|---|
ADE‑3416 | Addition of grantee type to CONFIG_ENTITY_PERMISSIONS to enable permissions to Snowflake SHAREs |
2.6 Deployment Management
New Features
ID | Description |
|---|---|
ADE‑3005 | New Deployment API |
ADE‑3513 | Jenkins job stylings based on job status |
ADE‑3324 | dh-metarepository-api: Out of memory from app (increase memory in Private AWS) |
Improvements
ID | Description |
|---|---|
ADE‑3317 | Investigate Jenkins lengthy pipeline processing for package deployments having NO_CHANGES |
ADE‑3320 | Improve update workflows in case of S3 to avoid unnecessary file transfers |
ADE‑3321 | Skip deployment run if newer one exists (special optimization) |
ADE‑3382 | Align names, titles and favicons with Deployment Management and Workflow Orchestration |
ADE‑3417 | Package import speed improvements |
ADE‑3463 | Disable system pre load job from Jenkins |
ADE‑3464 | Refactor: Gson to Jackson library and remove Metadata API DTO dependency |
ADE‑3465 | Change the "Time waiting in queue" in Jenkins jobs to zero seconds |
ADE‑3521 | Provide installation information also to the package job logs |
ADE‑3529 | Make package installation message more clear |
ADE‑3532 | Make package name in top breadcrumb bolder |
ADE‑3533 | Make package statuses bolder |
ADE‑3541 | Package deployment environment selection should indicate the action to be promotion |
ADE‑3542 | Change package build commitMessage to buildInfo and make it extend the actional commit message |
ADE‑3545 | Add visualization for deployment management |
ADE‑3562 | Update new favicon |
Fixes
ID | Description |
|---|---|
ADE‑3322 | Looks like metarepo api converter class used in import and export of packages may have request specific state and it still used as singleton |
ADE‑3407 | Manager StackOverflow with large pkg |
ADE‑3462 | Enable Hibernate's query plan cache parameters to avoid some of potential excessive memory usage problems |
ADE‑3638 | Double encoding with rest template |
ADE‑2925 | Showing logs in overlay views in jenkins are not working with theme colors at all |
ADE‑3307 | Package installation failure not descriptive in Jenkins |
ADE‑3318 | Use JWT originated author for commit instead of trusting client provided info in dto |
ADE‑3615 | Deploy error does not show the related SQL statement |
2.7 Loads and Workflows
Improvements
ID | Description |
|---|---|
ADE‑3392 | Cannot refer to relative paths with manifest and MS DATA SOURCES ie. Storage Integration |
ADE‑3382 | Align names, titles and favicons with Deployment Management and Workflow Orchestration |
ADE‑3403 | New token: <target_entity_logical_name> to be used in the loads |
ADE‑3559 | Notify API and Executor to support Parquet, Avro and ORC files in addition |
2.8 DBMS Support
New Features
ID | Description |
|---|---|
ADE‑3416 | Addition of grantee type to CONFIG_ENTITY_PERMISSIONS to enable permissions to Snowflake SHAREs |
ADE‑3187 | Force SSL for Redshift connections in Manager and Executor |
Fixes
ID | Description |
|---|---|
ADE‑3418 | Snowflake jdbc driver should be updated before 30th April 2021 |
ADE‑3552 | BigQuery JDBC driver update |
2.9 SaaS Related
APIs
ID | Description |
|---|---|
ADE‑2139 | External API Service Component |
ADE‑3609 | Notify API: Swagger theme improvements for markdown |
ADE‑3657 | Notify creation fails with aws s3 client error |
ADE‑3559 | Notify API and Executor to support Parquet, Avro and ORC files in addition |
Improvements
ID | Description |
|---|---|
ADE‑3198 | saas-installer to add elb access logging |
ADE‑3337 | Grafana user role to editor role |
ADE‑3421 | Theme update for Grafana 7.x |
ADE‑3660 | SaaS installer to support deployment-api migration mode |
Fixes
ID | Description |
|---|---|
ADE‑3658 | Modified user not shown in Designer |