File load bucket/container

Edition: Private

Agile Data Engine Private Edition installation creates a default cloud storage S3 bucket (AWS) or blob container (Azure) for incoming data per Runtime environment. Source data files can be placed in this cloud storage and loaded into the target database with file loads without additional authentication configuration between the target database and a storage bucket/container. Agile Data Engine will enter an identity and a key/token within data load commands for the target database to access files in the default storage.

See also:

• Manifest load

• Single file load

• Notifications

• Loading source data (guide)

Private AWS

File load bucket name is defined in the Runtime installation, buckets are created per Runtime environment. Please refer to your project documentation for the bucket names.

For example:

s3://example-file-load-bucket/

Data files can be uploaded into the bucket in a folder structure, for example:

s3://example-file-load-bucket/source_entity/filename

Manifests should be placed in a manifest folder with a similar folder structure:

s3://example-file-load-bucket/manifest/source_entity/manifestname.json

Private Azure

File load container name and storage account name are defined in the Runtime installation. Storage accounts and file load containers are created per Runtime environment. Please refer to your project documentation for the storage account and container names.

For example:

azure://examplestorageaccount.blob.core.windows.net/example-file-load-container/

Data files can be uploaded into the bucket in a folder structure, for example:

azure://examplestorageaccount.blob.core.windows.net/example-file-load-container/source_entity/filename

Manifests should be placed in a manifest folder with a similar folder structure:

azure://examplestorageaccount.blob.core.windows.net/example-file-load-container/manifest/source_entity/manifestname.json

Notes

IMPORTANT: Check the networking and security settings of the default storage service before uploading any data files there. Evaluate the need for additional security policies and network restrictions. Note that such policies and restrictions might require configuration in the target database to enable access to source files. The customer/project team is responsible for platform security in Agile Data Engine Private Edition installations.

Notification paths and filenames in the notification bucket/container have to match the file/manifest names and paths in the file load bucket/container.