Updating ADE Private Edition
Edition: Private
Guide objective:
This article guides you through the Agile Data Engine Private Edition release update process.
Note that the Agile Data Engine SaaS editions receive updates automatically as part of the service.
Prerequisites
Agile Data Engine release versions and related details are documented in the Release Notes. Please read the release notes carefully before starting the update process. Some releases might require additional manual steps.
Release updates are done per environment; it is advised to start with the least critical Runtime environment, to continue with other Runtime environments and lastly, to update the Designer.
Find a suitable window for a service break and inform the users as updating a Runtime environment requires pausing the workflows. Most release updates take less than an hour to complete. Confirm the estimated service break length and see that everything goes smoothly with the least critical Runtime environment.
Login to your Agile Data Engine installation environment (e.g. a Linux VM) and follow the instructions.
Runtimes
Repeat the following steps for each Runtime environment.
1. Pause all workflows
AWS
In AWS, workflows can be gracefully paused with an infrastructure maintenance script. The script waits that all tasks are done and stores the state of the workflow (paused/active). Run the script in the installation environment:
./run_infra_maintenance.sh {path-to-ade-main-configuration} {runtime-environment-name} DAGGER_STOP
For example:
./run_infra_maintenance.sh ../ade-main-conf.yaml dev DAGGER_STOP
Azure
Pause all active workflows manually in Workflow Orchestration ensuring that the tasks have finished first.
2. Take manual metadata backups
AWS
Create a database snapshot of the metadata repository Amazon RDS instance by following the AWS documentation.
Azure
Use backup scripts to take backups of the metadata databases. In the installation environment, go to the runtime specific target folder, e.g.
cd target/dev
Make sure that kubectl configuration is up to date:
./setup-kubectl.sh
Delete all pods to prevent anything from running:
kubectl delete deployments --all
Create backups. Note that there are multiple databases, metadata repository being the most critical one:
# Metadata repository
./run_job.sh backup-metarepository-db
# Grafana (if enabled in the environment)
./run_job.sh backup-grafana-db
# Workflow orchestration
./run_job.sh backup-dagger-db
You can list backups with:
./run_job.sh backup-list
Stay in the same target folder and also create Terraform backup with:
../../scripts/azure/terraform_state_backup.sh
3. Update main configuration release parameters
Edit the releaseVersion in the releaseParams block in the main configuration file to the target release version, for example:
adeEnvironments:
...
runtimes:
- name: dev
...
releaseParams:
releaseVersion: release-21_0_420_patch5
...
4. Execute the environment update
Load new package versions by running the update script with the packages-only option:
./update_ade_runtime.sh {path-to-ade-main-configuration} {runtime-environment-name} packages-only
For example:
./update_ade_runtime.sh ../ade-main-conf.yaml dev packages-only
Execute the environment update by running the update script with the skip-packages option, for example:
./update_ade_runtime.sh ../ade-main-conf.yaml dev skip-packages
5. Resume workflows
AWS
Resume paused workflows with the infrastructure maintenance script:
./run_infra_maintenance.sh {path-to-ade-main-configuration} {runtime-environment-name} DAGGER_START
For example:
./run_infra_maintenance.sh ../ade-main-conf.yaml dev DAGGER_START
Azure
Resume the workflows that were paused in step 1 manually in Workflow Orchestration.
Designer
1. Take manual metadata backups
AWS
Create a database snapshot of the metadata repository Amazon RDS instance by following the AWS documentation.
Azure
Use backup scripts to take backups of the metadata databases. In the installation environment, go to design target folder, e.g.
cd target/design
Make sure that kubectl configuration is up to date:
./setup-kubectl.sh
Delete all pods to prevent anything from running:
kubectl delete deployments --all
Create backups. Note that there are multiple databases, metadata repository being the most critical one:
# Metadata repository
./run_job.sh backup-metarepository-db
# Deployment API
./run_job.sh backup-deployment-api-db
# Grafana (if enabled in the environment)
./run_job.sh backup-grafana-db
You can list backups with:
./run_job.sh backup-list
Stay in the same target folder and also create Terraform backup with:
../../scripts/azure/terraform_state_backup.sh
2. Update main configuration release parameters
Edit the releaseVersion in the releaseParams block in the main configuration file to the target release version, for example:
adeEnvironments:
...
design:
- name: design
...
releaseParams:
releaseVersion: release-21_0_420_patch5
...
3. Execute the environment update
Load new package versions by running the update script with the packages-only option:
./update_ade_design.sh {path-to-ade-main-configuration} {design-environment-name} packages-only
For example:
./update_ade_design.sh ../ade-main-conf.yaml design packages-only
Execute the environment update by running the update script with the skip-packages option, for example:
./update_ade_design.sh ../ade-main-conf.yaml design skip-packages
Environment variables
Environment variables can be defined as follows:
export {environment-variable-name}={value}
For example:
export ADE_TERRAFORM_APPLY_SKIP=true
AWS
Variable | Type | Description |
---|---|---|
ADE_SKIP_AWS_CF_VALIDATION | true/false | If true, CloudFormation template validation is skipped. This can speed up the update process. |
Azure
Variable | Type | Description |
---|---|---|
ADE_TERRAFORM_APPLY_SKIP | true/false | If true, the environment update scripts will skip the Terraform apply step and no changes will be made. This variable can be used with the environment update scripts to create the scripting environment only, for example, if the backup scripts failed to run. |
Notes
Current release versions are specified in the main configuration file and they are also visible in the Designer front page under Environments.
Changing Runtime environment deployment settings (see Deployment Management) requires running the Design environment update as Deployment Management is part of the Design environment.
In Azure, services can be restarted with:
target/{environment-name}/restart-services.sh
Optionally, timeout in seconds can be specified as a parameter (default: 300s).