DAG generation mode
DAG generation mode controls how workflows are generated. It can be set on Runtime environment level with CONFIG_ENVIRONMENT_VARIABLES and on schedule level with CONFIG_LOAD_SCHEDULES.
See also:
List of DAG generation modes
Mode | Description |
---|---|
ENTITY_ORIENTED | Generated DAGs show entity operators making them easier to read. This used to be the default mode. However, it is not recommended due to the overhead from unnecessary operators. |
LOAD_ORIENTED | Generated DAGs contain only load tasks reducing the unnecessary scheduling/running overhead related to entity operators. |
OPTIMIZED_ENTITY_ORIENTED | Loads are executed in entity operators in a sequential manner. I.e. the DAGs contain entity operators and loads into those entities are executed within the entity operators. Only entities that have incoming loads are included. This mode prevents loads from blocking each other as multiple loads into the same target cannot be run parallel. This optimizes execution wait times caused by entity locks which consume pool slots when loads are queuing without actually doing anything. As a possible downside, loads are not visually separate in the DAGs. When opening a failed entity node you need to search for the actual failure in the log in case there are multiple loads into that entity. |
OPTIMIZED_LOAD_ORIENTED | Similar to LOAD_ORIENTED, but further excludes unnecessary tasks, e.g. disabled loads or loads that do not have any commands. |
SEQUENTIAL_TASK_ORIENTED | A special mode for getting rid of overhead in simple DAGs meant to run in a fast cycle (near realtime). Generated DAGs only contain one task which runs all loads in sequence. The running order of loads which are not dependent on each other is random but the original order of dependent loads is followed. This mode disables all parallelism inside the DAG, therefore it is advised to carefully test it case by case before enabling it. Do not set the mode with an environment variable, instead set it on schedule level with CONFIG_LOAD_SCHEDULES. |
Notes
OPTIMIZED_ENTITY_ORIENTED or OPTIMIZED_LOAD_ORIENTED is recommended as the default setting.
Schedule level DAG generation mode settings override environment level settings.