Triggering a Workflow
A workflow can be triggered by:
A schedule defined by a cron expression in CONFIG_LOAD_SCHEDULES
Manual triggering in Workflow Orchestration
An API request (SaaS: see External API)
Another worfklow or by itself (see CONFIG_LOAD_SCHEDULES)
Workflows can be executed manually with parameters in Workflow Orchestration by pressing Trigger DAG. This might be useful especially in the development phase. See details on how to set the parameters below.
See also:
Tutorials
See the below videos for quick tutorials on how to load data by triggering a workflow and check the load statuses and logs.
https://youtu.be/_0EUDTFO9Dkhttps://youtu.be/e0Ks6VEWTe8
Triggering a workflow manually with parameters
The following parameters are available:
Parameter | Type | Example | Notes |
|---|---|---|---|
on_error_pause_dag | Boolean | {"on_error_pause_dag": true} | If a task fails, DAG will be paused. Defaults to schedule configuration. |
load | string | {"load": "load_name_to_run"} | Only the defined load will be run. No effect if the value is empty. |
loads | string array | {"loads": ["load_name_to_run", "load_name_to_run_2"]} | Only the defined loads will be run. No effect if the value is empty. |
package | string | {"package": "package_of_loads_to_run"} | Only loads in the defined package will be run. No effect if the value is empty. |
packages | string array | {"packages": ["package_of_loads_to_run", "package_of_loads_to_run_2"]} | Only loads in the defined packages will be run. No effect if the value is empty. |
trigger_workflows | string array | {"trigger_workflows":["WORKFLOW_TO_TRIGGER"]} | Only the defined workflows will be triggered after workflow execution. Note that this can only be used to prevent existing triggers in the schedule configuration (CONFIG_LOAD_SCHEDULES) from triggering. You cannot configure additional workflow triggers here that are not defined in the schedule configuration. |
You can combine parameters in a single JSON. However, if both load and loads are defined, only load will be used. Same for package and packages.
To trigger a workflow manually, press Trigger DAG in Workflow Orchestration. Optionally set the CONFIGURATION JSON:
Notes
Workflow(s) need to be set active in Workflow Orchestration with the toggle switch next to the workflow name. Paused workflows will not be scheduled.
Examples
Triggering manually with parameters
Parameters set when manually triggering workflow TRIPDATA:
{
"loads": ["load_d_ratecode_from_data_vault", "load_f_trip_from_data_vault"],
"on_error_pause_dag": true
}Only the defined loads will be executed and the workflow will be paused on error.