OPT_USE_RUN_IDS_INCREMENTALLY

OPT_USE_RUN_IDS_INCREMENTALLY is a load option that improves the performance of Agile Data Engine internal metadata queries in certain situations when using Run ID logic.

With OPT_USE_RUN_IDS_INCREMENTALLY = true, new run ids will only be processed from the previous load time onwards; i.e. Agile Data Engine will use the latest logged run timestamp of the load when querying loadable run ids internally.

See also:

Usage

OPTION

OPT_USE_RUN_IDS_INCREMENTALLY: boolean

Default: false

Notes

OPT_USE_RUN_IDS_INCREMENTALLY is useful in cases where a lot of run ids are produced such as near real-time cases with lots of files and file specific run ids.

Use with:

OPTION

OPT_USE_RUN_IDS: true

Like Run ID logic in general, OPT_USE_RUN_IDS_INCREMENTALLY can be used in table loads that have a single entity mapping (i.e. one source entity).

Examples

Using OPT_USE_RUN_IDS_INCREMENTALLY

See examples on using Run ID logic in OPT_USE_RUN_IDS.

Enable Run ID logic for a load:

OPTION

OPT_USE_RUN_IDS: true

Set run ids to be processed incrementally:

OPTION

OPT_USE_RUN_IDS_INCREMENTALLY: true

When the load is executed, Agile Data Engine will use the timestamp of the previous logged execution of the load when querying internal metadata for new run ids.

For example, if the load was previously run at '2022-07-11T11:48:30.784000', the load log would display:

LOG

INFO - OPT_USE_RUN_IDS_INCREMENTALLY is set, fetching RunIds from: 2022-07-11T11:48:30.784000