Skip to main content
Skip table of contents

Updating ADE Private Edition

Edition: Private

Guide objective:
This article guides you through the Agile Data Engine Private Edition release update process.

Note that the Agile Data Engine SaaS editions receive updates automatically as part of the service.


Prerequisites

Agile Data Engine release versions and related details are documented in the Release Notes. Please read the release notes carefully before starting the update process. Some releases might require additional manual steps.

Release updates are done per environment; it is advised to start with the least critical Runtime environment, to continue with other Runtime environments and lastly, to update the Designer.

Find a suitable window for a service break and inform the users as updating a Runtime environment requires pausing the workflows. Most release updates take less than an hour to complete. Confirm the estimated service break length and see that everything goes smoothly with the least critical Runtime environment.

Login to your Agile Data Engine installation environment (e.g. a Linux VM) and follow the instructions.


Runtimes

Repeat the following steps for each Runtime environment.

1. Pause all workflows

AWS

In AWS, workflows can be gracefully paused with an infrastructure maintenance script. The script waits that all tasks are done and stores the state of the workflow (paused/active). Run the script in the installation environment:

BASH
./run_infra_maintenance.sh {path-to-ade-main-configuration} {runtime-environment-name} DAGGER_STOP

For example:

BASH
./run_infra_maintenance.sh ../ade-main-conf.yaml dev DAGGER_STOP
Azure

Pause all active workflows manually in Workflow Orchestration ensuring that the tasks have finished first.

2. Take manual metadata backups

AWS

Create a database snapshot of the metadata repository Amazon RDS instance by following the AWS documentation.

Azure

Use backup scripts to take backups of the metadata databases. In the installation environment, go to the runtime specific target folder, e.g.

BASH
cd target/dev

Make sure that kubectl configuration is up to date:

BASH
./setup-kubectl.sh

Delete all pods to prevent anything from running:

BASH
kubectl delete deployments --all

Create backups. Note that there are multiple databases, metadata repository being the most critical one:

BASH
# Metadata repository
./run_job.sh backup-metarepository-db

# Grafana (if enabled in the environment)
./run_job.sh backup-grafana-db

# Workflow orchestration
./run_job.sh backup-dagger-db

You can list backups with:

BASH
./run_job.sh backup-list

Stay in the same target folder and also create Terraform backup with:

BASH
../../scripts/azure/terraform_state_backup.sh

3. Update main configuration release parameters

Edit the releaseVersion in the releaseParams block in the main configuration file to the target release version, for example:

YAML
adeEnvironments:
...
  runtimes:
  - name: dev
    ...
    releaseParams:
      releaseVersion: release-21_0_420_patch5
...

4. Execute the environment update

Load new package versions by running the update script with the packages-only option:

BASH
./update_ade_runtime.sh {path-to-ade-main-configuration} {runtime-environment-name} packages-only

For example:

BASH
./update_ade_runtime.sh ../ade-main-conf.yaml dev packages-only

Execute the environment update by running the update script with the skip-packages option, for example:

BASH
./update_ade_runtime.sh ../ade-main-conf.yaml dev skip-packages

5. Resume workflows

AWS

Resume paused workflows with the infrastructure maintenance script:

BASH
./run_infra_maintenance.sh {path-to-ade-main-configuration} {runtime-environment-name} DAGGER_START

For example:

BASH
./run_infra_maintenance.sh ../ade-main-conf.yaml dev DAGGER_START
Azure

Resume the workflows that were paused in step 1 manually in Workflow Orchestration.


Designer

1. Take manual metadata backups

AWS

Create a database snapshot of the metadata repository Amazon RDS instance by following the AWS documentation.

Azure

Use backup scripts to take backups of the metadata databases. In the installation environment, go to design target folder, e.g.

BASH
cd target/design

Make sure that kubectl configuration is up to date:

BASH
./setup-kubectl.sh

Delete all pods to prevent anything from running:

BASH
kubectl delete deployments --all

Create backups. Note that there are multiple databases, metadata repository being the most critical one:

BASH
# Metadata repository
./run_job.sh backup-metarepository-db

# Deployment API
./run_job.sh backup-deployment-api-db

# Grafana (if enabled in the environment)
./run_job.sh backup-grafana-db

You can list backups with:

BASH
./run_job.sh backup-list

Stay in the same target folder and also create Terraform backup with:

BASH
../../scripts/azure/terraform_state_backup.sh

2. Update main configuration release parameters

Edit the releaseVersion in the releaseParams block in the main configuration file to the target release version, for example:

YAML
adeEnvironments:
...
  design:
  - name: design
    ...
    releaseParams:
      releaseVersion: release-21_0_420_patch5
...

3. Execute the environment update

Load new package versions by running the update script with the packages-only option:

BASH
./update_ade_design.sh {path-to-ade-main-configuration} {design-environment-name} packages-only

For example:

BASH
./update_ade_design.sh ../ade-main-conf.yaml design packages-only

Execute the environment update by running the update script with the skip-packages option, for example:

BASH
./update_ade_design.sh ../ade-main-conf.yaml design skip-packages

Environment variables

Environment variables can be defined as follows:

BASH
export {environment-variable-name}={value}

For example:

BASH
export ADE_TERRAFORM_APPLY_SKIP=true

AWS

Variable

Type

Description

ADE_SKIP_AWS_CF_VALIDATION

true/false

If true, CloudFormation template validation is skipped. This can speed up the update process.

Azure

Variable

Type

Description

ADE_TERRAFORM_APPLY_SKIP

true/false

If true, the environment update scripts will skip the Terraform apply step and no changes will be made. This variable can be used with the environment update scripts to create the scripting environment only, for example, if the backup scripts failed to run.

Notes

Current release versions are specified in the main configuration file and they are also visible in the Designer front page under Environments.

Changing Runtime environment deployment settings (see Deployment Management) requires running the Design environment update as Deployment Management is part of the Design environment.

In Azure, services can be restarted with:

BASH
target/{environment-name}/restart-services.sh

Optionally, timeout in seconds can be specified as a parameter (default: 300s).

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.