OPT_USE_RUN_IDS_INCREMENTALLY is a load option that improves the performance of Agile Data Engine internal metadata queries in certain situations when using Run ID logic.
With OPT_USE_RUN_IDS_INCREMENTALLY = true, new run ids will only be processed from the previous load time onwards; i.e. Agile Data Engine will use the latest logged run timestamp of the load when querying loadable run ids internally.
See also:
Usage
OPT_USE_RUN_IDS_INCREMENTALLY: boolean
Default: false
Notes
OPT_USE_RUN_IDS_INCREMENTALLY is useful in cases where a lot of run ids are produced such as near real-time cases with lots of files and file specific run ids.
Use with:
OPT_USE_RUN_IDS: true
Like Run ID logic in general, OPT_USE_RUN_IDS_INCREMENTALLY can be used in table loads that have a single entity mapping (i.e. one source entity).
Examples
Using OPT_USE_RUN_IDS_INCREMENTALLY
See examples on using Run ID logic in OPT_USE_RUN_IDS.
Enable Run ID logic for a load:
OPT_USE_RUN_IDS: true
Set run ids to be processed incrementally:
OPT_USE_RUN_IDS_INCREMENTALLY: true
When the load is executed, Agile Data Engine will use the timestamp of the previous logged execution of the load when querying internal metadata for new run ids.
For example, if the load was previously run at '2022-07-11T11:48:30.784000', the load log would display:
INFO - OPT_USE_RUN_IDS_INCREMENTALLY is set, fetching RunIds from: 2022-07-11T11:48:30.784000