Triggering a Workflow

A workflow can be triggered by:

A schedule defined by a cron expression in CONFIG_LOAD_SCHEDULES
Manual triggering in Workflow Orchestration
An API request (SaaS: see External API)
Another worfklow or by itself (see CONFIG_LOAD_SCHEDULES)

Workflows can be executed manually with parameters in Workflow Orchestration by pressing Trigger DAG. This might be useful especially in the development phase. See details on how to set the parameters below.

See also:

Tutorials

See the below videos for quick tutorials on how to load data by triggering a workflow and check the load statuses and logs.

https://youtu.be/_0EUDTFO9Dk

https://youtu.be/e0Ks6VEWTe8

Triggering a workflow manually with parameters

The following parameters are available:

Parameter	Type	Example	Notes
on_error_pause_dag	Boolean	{"on_error_pause_dag": true}	If a task fails, DAG will be paused. Defaults to schedule configuration.
load	string	{"load": "load_name_to_run"}	Only the defined load will be run. No effect if the value is empty.
loads	string array	{"loads": ["load_name_to_run", "load_name_to_run_2"]}	Only the defined loads will be run. No effect if the value is empty.
package	string	{"package": "package_of_loads_to_run"}	Only loads in the defined package will be run. No effect if the value is empty.
packages	string array	{"packages": ["package_of_loads_to_run", "package_of_loads_to_run_2"]}	Only loads in the defined packages will be run. No effect if the value is empty.
trigger_workflows	string array	{"trigger_workflows":["WORKFLOW_TO_TRIGGER"]}	Only the defined workflows will be triggered after workflow execution. Note that this can only be used to prevent existing triggers in the schedule configuration (CONFIG_LOAD_SCHEDULES) from triggering. You cannot configure additional workflow triggers here that are not defined in the schedule configuration.

You can combine parameters in a single JSON. However, if both load and loads are defined, only load will be used. Same for package and packages.

To trigger a workflow manually, press Trigger DAG in Workflow Orchestration. Optionally set the CONFIGURATION JSON:

In 24.x version of Agile Data Engine user can give the parameters either in input fields or in Generated Configuration JSON section. For passing multiple of load names pass the values on separate lines in the LOADS-input field.

Notes

Workflow(s) need to be set active in Workflow Orchestration with the toggle switch next to the workflow name. Paused workflows will not be scheduled.

Examples

Triggering manually with parameters

Parameters set when manually triggering workflow TRIPDATA:

JSON

{
    "loads": ["load_d_ratecode_from_data_vault", "load_f_trip_from_data_vault"],
    "package": null,
    "on_error_pause_dag": false,
    "trigger_workflows": null
}

Only the defined loads will be executed and the workflow will be paused on error.