Updating ADE Private Edition

Edition: Private

Guide objective:
This article guides you through the Agile Data Engine Private Edition release update process.

Note that the Agile Data Engine SaaS editions receive updates automatically as part of the service.

Prerequisites

Agile Data Engine release versions and related details are documented in the Release Notes. Please read the release notes carefully before starting the update process. Some releases might require additional manual steps.

Release updates are done per environment; it is advised to start with the least critical Runtime environment, to continue with other Runtime environments and lastly, to update the Designer.

Find a suitable window for a service break and inform the users as updating a Runtime environment requires pausing the workflows. Most release updates take less than an hour to complete. Confirm the estimated service break length and see that everything goes smoothly with the least critical Runtime environment.

Login to your Agile Data Engine installation environment (e.g. a Linux VM) and follow the instructions.

Runtimes

Repeat the following steps for each Runtime environment.

1. Pause all workflows

AWS

In AWS, workflows can be gracefully paused with an infrastructure maintenance script. The script waits that all tasks are done and stores the state of the workflow (paused/active). Run the script in the installation environment:

BASH

./run_infra_maintenance.sh {path-to-ade-main-configuration} {runtime-environment-name} DAGGER_STOP

For example:

BASH

./run_infra_maintenance.sh ../ade-main-conf.yaml dev DAGGER_STOP

Azure

Pause all active workflows manually in Workflow Orchestration ensuring that the tasks have finished first.

2. Take manual metadata backups

AWS

Create a database snapshot of the metadata repository Amazon RDS instance by following the AWS documentation.

Azure

Use backup scripts to take backups of the metadata databases. In the installation environment, go to the runtime specific target folder, e.g.

BASH

cd target/dev

Make sure that kubectl configuration is up to date:

BASH

./setup-kubectl.sh

Delete all pods to prevent anything from running:

BASH

kubectl delete deployments --all

Create backups. Note that there are multiple databases, metadata repository being the most critical one:

BASH

# Metadata repository
./run_job.sh backup-metarepository-db

# Grafana (if enabled in the environment)
./run_job.sh backup-grafana-db

# Workflow orchestration
./run_job.sh backup-dagger-db

You can list backups with:

BASH

./run_job.sh backup-list

Stay in the same target folder and also create Terraform backup with:

BASH

../../scripts/azure/terraform_state_backup.sh

3. Update main configuration release parameters

Edit the releaseVersion in the releaseParams block in the main configuration file to the target release version, for example:

YAML

adeEnvironments:
...
  runtimes:
  - name: dev
    ...
    releaseParams:
      releaseVersion: release-21_0_420_patch5
...

4. Execute the environment update

Load new package versions by running the update script with the packages-only option:

BASH

./update_ade_runtime.sh {path-to-ade-main-configuration} {runtime-environment-name} packages-only

For example:

BASH

./update_ade_runtime.sh ../ade-main-conf.yaml dev packages-only

Execute the environment update by running the update script with the skip-packages option, for example:

BASH

./update_ade_runtime.sh ../ade-main-conf.yaml dev skip-packages

5. Resume workflows

AWS

Resume paused workflows with the infrastructure maintenance script:

BASH

./run_infra_maintenance.sh {path-to-ade-main-configuration} {runtime-environment-name} DAGGER_START

For example:

BASH

./run_infra_maintenance.sh ../ade-main-conf.yaml dev DAGGER_START

Azure

Resume the workflows that were paused in step 1 manually in Workflow Orchestration.

Designer

1. Take manual metadata backups

AWS

Create a database snapshot of the metadata repository Amazon RDS instance by following the AWS documentation.

Azure

Use backup scripts to take backups of the metadata databases. In the installation environment, go to design target folder, e.g.

BASH

cd target/design

Make sure that kubectl configuration is up to date:

BASH

./setup-kubectl.sh

Delete all pods to prevent anything from running:

BASH

kubectl delete deployments --all

Create backups. Note that there are multiple databases, metadata repository being the most critical one:

BASH

# Metadata repository
./run_job.sh backup-metarepository-db

# Deployment API
./run_job.sh backup-deployment-api-db

# Grafana (if enabled in the environment)
./run_job.sh backup-grafana-db

You can list backups with:

BASH

./run_job.sh backup-list

Stay in the same target folder and also create Terraform backup with:

BASH

../../scripts/azure/terraform_state_backup.sh

2. Update main configuration release parameters

Edit the releaseVersion in the releaseParams block in the main configuration file to the target release version, for example:

YAML

adeEnvironments:
...
  design:
  - name: design
    ...
    releaseParams:
      releaseVersion: release-21_0_420_patch5
...

3. Execute the environment update

Load new package versions by running the update script with the packages-only option:

BASH

./update_ade_design.sh {path-to-ade-main-configuration} {design-environment-name} packages-only

For example:

BASH

./update_ade_design.sh ../ade-main-conf.yaml design packages-only

Execute the environment update by running the update script with the skip-packages option, for example:

BASH

./update_ade_design.sh ../ade-main-conf.yaml design skip-packages

Environment variables

Environment variables can be defined as follows:

BASH

export {environment-variable-name}={value}

For example:

BASH

export ADE_TERRAFORM_APPLY_SKIP=true

AWS

Variable	Type	Description
ADE_SKIP_AWS_CF_VALIDATION	true/false	If true, CloudFormation template validation is skipped. This can speed up the update process.

Azure

Variable	Type	Description
ADE_TERRAFORM_APPLY_SKIP	true/false	If true, the environment update scripts will skip the Terraform apply step and no changes will be made. This variable can be used with the environment update scripts to create the scripting environment only, for example, if the backup scripts failed to run.

Notes

Current release versions are specified in the main configuration file and they are also visible in the Designer front page under Environments.

Changing Runtime environment deployment settings (see Deployment Management) requires running the Design environment update as Deployment Management is part of the Design environment.

In Azure, services can be restarted with:

BASH

target/{environment-name}/restart-services.sh

Optionally, timeout in seconds can be specified as a parameter (default: 300s).