OPT_USE_RUN_IDS_INCREMENTALLY is a load option that improves the performance of Agile Data Engine internal metadata queries in certain situations when using Run ID logic.
With OPT_USE_RUN_IDS_INCREMENTALLY = true, new run ids will only be processed from the previous load time onwards; i.e. Agile Data Engine will use the latest logged run timestamp of the load when querying loadable run ids internally.
OPT_USE_RUN_IDS_INCREMENTALLY is useful in cases where a lot of run ids are produced such as near real-time cases with lots of files and file specific run ids.
Like Run ID logic in general, OPT_USE_RUN_IDS_INCREMENTALLY can be used in table loads that have a single entity mapping (i.e. one source entity).
See examples on using Run ID logic in OPT_USE_RUN_IDS.
Enable Run ID logic for a load:
Set run ids to be processed incrementally:
When the load is executed, Agile Data Engine will use the timestamp of the previous logged execution of the load when querying internal metadata for new run ids.
For example, if the load was previously run at '2022-07-11T11:48:30.784000', the load log would display:
INFO - OPT_USE_RUN_IDS_INCREMENTALLY is set, fetching RunIds from: 2022-07-11T11:48:30.784000