OPT_DATA_FILE_LOCATION

Edition: Private

Target database: Snowflake, Google BigQuery

OPT_DATA_FILE_LOCATION is a load option that overrides the data file location in a single file load.

See also:

Usage

OPTION

OPT_DATA_FILE_LOCATION: string

Notes

OPT_DATA_FILE_LOCATION can be used with file loads when loading single files. With manifest loading, OPT_DATA_FILE_LOCATION is not required as file locations will be given in the manifest.

When a file location is given with OPT_DATA_FILE_LOCATION, notified files will be loaded from the given location instead of loading from Agile Data Engine default file load bucket/container.

Use OPT_STORAGE_INTEGRATION to provide authorization to the file location.

Examples

Snowflake file load

File location given with OPT_DATA_FILE_LOCATION and authorization with OPT_STORAGE_INTEGRATION:

OPTION

OPT_DATA_FILE_LOCATION: 
azure://adetaxidata.blob.core.windows.net/test-data

OPT_STORAGE_INTEGRATION:
ade_taxi_data

File load executed in Snowflake:

SQL

COPY INTO ...
FROM 'azure://adetaxidata.blob.core.windows.net/test-data/test_src/table.test_src.1516355232334.batch.1516355232334.fullscanned.false.delim.semicolon.csv'
STORAGE_INTEGRATION=ade_taxi_data
FILE_FORMAT=(...);

Google BigQuery file load

File location given with OPT_DATA_FILE_LOCATION and authorization with OPT_STORAGE_INTEGRATION:

OPTION

OPT_DATA_FILE_LOCATION: 
gs://test-data

OPT_STORAGE_INTEGRATION:
ade_taxi_data

File load executed in Google BigQuery:

SQL

LOAD DATA INTO staging_bq.test
FROM FILES(
  format='CSV',
  uris = ['gs://test-data/test_src/table.test_src.1516355232334.batch.1516355232334.fullscanned.false.delim.semicolon.csv']
)
WITH CONNECTION ade_taxi_data;