Configuration of Database Tags
Database Tags in Snowflake
Precondition on Snowflake
Tags are required to be defined on target database. Defining their allowed values are optional.
Do notice that tags are schema specific.
Database tag configuration:
Entity level TAG
In example we are creating a tag configuration Cost Center with values marketing and development.
-- Define TAG in Snowflake (loose definition):
create tag tags.cost_center;
-- or more restrictively:
create tag tags.cost_center allowed_values 'marketing','development';
Grant APPLY TAG permission for the row Agile Data Engine is using
Open CONFIG_ENTITY_DEFAULTS - package in Agile Data Engine
Open Configuration
Choose Database Tags - tab
Click Create Database Tag
Create Database Tag with following information
Label - Label for the Tag shown in Agile Data Engine, for example “Cost Center”
Technical Name - Schema specific TAG name in Snowflake “tags.cost_center”
Description - Short description for which purposes this tag is to be used for
Select options:
Allowed on entity level
has values
Click “Create”
Click the name of freshly created Cost Center from the Database tag list
Click Create Database Tag Value
Type in the wanted Tag value information (marketing)
Value - Tag value in Snowflake
Description - Short description for the tag value
Click “Create”
Click Create Database Tag Value
Type in the wanted Tag value information (development)
Value - Tag value in Snowflake
Description - Short description for the tag value
Click “Create”
Commit the CONFIG_ENTITY_DEFAULT package
Attribute level TAG
-- Define TAG in Snowflake (loose definition):
create tag tags.information_type;
-- or more restrictively:
create tag tags.information_type allowed_values 'pii','nationality';
Grant APPLY TAG permission for the row Agile Data Engine is using
Open CONFIG_ENTITY_DEFAULTS - package in Agile Data Engine
Open Configuration
Choose Database Tags - tab
Click Create Database Tag
Create Database Tag with following information
Label - Label for the Tag shown in Agile Data Engine, for example “Information type”
Technical Name - Schema specific TAG name in Snowflake “tags.information_type”
Description - Short description for which purposes this tag is to be used for
Select options:
Allowed on attribute level
has values
Click “Create”
Click the name of freshly created Information type from the Database tag list
Click Create Database Tag Value
Type in the wanted Tag value information (pii)
Value - Tag value in Snowflake
Description - Short description for the tag value
Click “Create”
Click Create Database Tag Value
Type in the wanted Tag value information (nationality)
Value - Tag value in Snowflake
Description - Short description for the tag value
Click “Create”
Commit the CONFIG_ENTITY_DEFAULT package
Generated DDL
DDL generated by Agile Data Engine on Entity level:
ALTER TABLE data SET TAG cost_center = 'marketing';
DDL generated by Agile Data Engine on Attribute level:
ALTER TABLE data ALTER COLUMN phone_number SET TAG information_type = 'pii';
Database Tags in Databricks SQL
Database tag configuration:
Entity level TAG
In example we are creating a tag configuration Cost Center with values marketing and development.
Open CONFIG_ENTITY_DEFAULTS - package in Agile Data Engine
Open Configuration
Choose Database Tags - tab
Click Create Database Tag
Create Database Tag with following information
Label - Label for the Tag shown in Agile Data Engine, for example “Cost Center”
Technical Name - TAG name in Databricks “cost_center”
Description - Short description for which purposes this tag is to be used for
Select options:
Allowed on entity level
has values
Click “Create”
Click the name of freshly created Cost Center from the Database tag list
Click Create Database Tag Value
Type in the wanted Tag value information (marketing)
Value - Tag value in Databricks
Description - Short description for the tag value
Click “Create”
Click Create Database Tag Value
Type in the wanted Tag value information (development)
Value - Tag value in Databricks
Description - Short description for the tag value
Click “Create”
Commit the CONFIG_ENTITY_DEFAULT package
Attribute level TAG
Open CONFIG_ENTITY_DEFAULTS - package in Agile Data Engine
Open Configuration
Choose Database Tags - tab
Click Create Database Tag
Create Database Tag with following information
Label - Label for the Tag shown in Agile Data Engine, for example “Information type”
Technical Name - TAG name in Databricks “information_type”
Description - Short description for which purposes this tag is to be used for
Select options:
Allowed on attribute level
has values
Click “Create”
Click the name of freshly created Information type from the Database tag list
Click Create Database Tag Value
Type in the wanted Tag value information (pii)
Value - Tag value in Databricks
Description - Short description for the tag value
Click “Create”
Click Create Database Tag Value
Type in the wanted Tag value information (nationality)
Value - Tag value in Databricks
Description - Short description for the tag value
Click “Create”
Commit the CONFIG_ENTITY_DEFAULT package
Generated DDL
DDL generated by Agile Data Engine on Entity level:
ALTER TABLE data SET TAGS ('cost_center' = 'marketing');
DDL generated by Agile Data Engine on Attribute level:
ALTER TABLE data ALTER COLUMN phone_number SET TAGS ('information_type' = 'pii');
Database Tags in Google BigQuery
Precondition on BigQuery
Resource Tags are required to be defined on target project (https://console.cloud.google.com/iam-admin/tags). These affect on the entity level and only to TABLEs.
https://cloud.google.com/bigquery/docs/tags
Policy Tags are required to be defined on target project https://console.cloud.google.com/bigquery/policy-tags. These affect on the attribute level. No notice that only one Policy Tag is allowed per attribute.
Entity level TAG (Resource Tag)
In example we are creating a tag configuration Cost Center with values marketing and development.
Open CONFIG_ENTITY_DEFAULTS - package in Agile Data Engine
Open Configuration
Choose Database Tags - tab
Click Create Database Tag
Create Database Tag with following information
Label - Label for the Tag shown in Agile Data Engine, for example “Cost Center”
Technical Name - TAG name in BigQuery having the project id included
CODE<project-id>/cost_center
Description - Short description for which purposes this tag is to be used for
Select options:
Allowed on entity level
has values
Click “Create”
Click the name of freshly created Cost Center from the Database tag list
Click Create Database Tag Value
Type in the wanted Tag value information (marketing)
Value - Tag value in BigQuery
Description - Short description for the tag value
Click “Create”
Click Create Database Tag Value
Type in the wanted Tag value information (development)
Value - Tag value in BigQuery
Description - Short description for the tag value
Click “Create”
Commit the CONFIG_ENTITY_DEFAULT package
Attribute level TAG (Policy Tag)
Open CONFIG_ENTITY_DEFAULTS - package in Agile Data Engine
Open Configuration
Choose Database Tags - tab
Click Create Database Tag
Create Database Tag with following information
Label - Label for the Tag shown in Agile Data Engine, for example “Information type”
Technical Name - The ID of the Policy Tag in BigQuery, syntax is of type
CODEprojects/<project_id>/locations/<location>/taxonomies/<taxonomy_id>/policyTags/<policy_tag_id>
Description - Short description for which purposes this tag is to be used for
Select options:
Allowed on attribute level
Click “Create”
Commit the CONFIG_ENTITY_DEFAULT package
Entity level Resource Tags will be updated through BigQuery API call, where the table involved will be updated to contain the desired resource tags.
Attribute level Policy Tags will be updated through BigQuery API call, where the table schema involved will be updated to contain the desired policy tags on desired columns
Required permissions in Google BigQuery
For ADE service account, the following permissions need to be added:
-- Permissions to bind or unbind tags
resourcemanager.tagValueBindings.create
resourcemanager.tagValueBindings.delete
-- Permissions to get taxonomies and assign those to columns
datacatalog.taxonomies.get
bigquery.tables.setCategory
-- Permissions to query masked data
bigquery.dataPolicies.maskedGet
datacatalog.categories.fineGrainedGet