Workflow Orchestration

Workflow Orchestration controls the execution of load workflows also known as DAGs (directed acyclic graph). Workflow Orchestration is a component of Runtime environments, i.e. each Runtime environment has a separate Workflow Orchestration.

See also:

• CONFIG_LOAD_SCHEDULES

• Loads

• Entity mapping

Tutorials

See the below video for a quick tutorial on how to check the load statuses and logs.

Workflow generation

Workflows are generated automatically based on entity dependencies defined in load entity mappings, and on schedules defined in CONFIG_LOAD_SCHEDULES and selected in the loads. When a schedule is selected for a load in an entity, all loads of dependent entities are automatically placed in that schedule unless overridden by a manual load selection.

Load order in a workflow is also determined from the entity dependencies mapped in entity mappings. Independent entities can be loaded in parallel; the level of parallelism is limited by the pool size, by the number of other workflows running simultaneously within the same pool, and by the resources of the target database.

DAG generation mode determines how a workflow is generated. It can impact performance and can also affect parallelism.

Managing workflows

Workflows are listed in the DAGs view in Workflow Orchestration which contains a toggle switch to enable/disable a workflow and the basic information of each workflow in that Runtime environment.

A single DAG view is opened by pressing the name of a workflow. It contains, for example, the

• Graph view which is a graphical representation of the workflow. While the workflow is running, statuses are shown per task. Load logs can be accessed by clicking a task name and selecting View Log.

• Tree view which shows the current and previous executions of the workflow including statuses. Load logs can be accessed by clicking a task execution (box) and selecting View Log.

• Task Duration, Landing Times and Gantt show various statistics of workflow executions.

• Trigger DAG can be used for manually triggering a workflow.

• Refresh updates the information e.g. when workflows are running.

Notes

Do not use the Mark Failed or Mark Success options to cancel tasks or workflows. Agile Data Engine acquires internal entity locks for entities being processed which are not cleared if these features are used. Instead, e.g. long running loads should be aborted from the target database. This will mark the task as failed and entity locks will be cleared.

By default, locks are cleared in 90 minutes. Also see OPT_LOCK_WAIT_TIME_IN_SECS.