Using Metadata Interface

Guide objective

This guide explains how to use the Metadata Interface.

Edition: SaaS Enterprise

The Metadata Interface is used to query metadata of your Agile Data Engine tenant. It contains metadata about packages, entities, loads, schedules, workflows etc. Use External API to connect the Metadata Interface. For test and development purposes, you can use the Swagger UI or a GraphQL client such as GraphQL Playground or Postman.

Metadata Interface Data Model

An “indicative“ ER model of the Metadata Interface can be found below. Remember to check the model in the Metadata Interface itself, because the model may vary slightly between different versions.

Metadata in Design & Runtime environments

Note that the available metadata depends on environment:

• Design environment (DESIGN)

  • Entity & package metadata as seen in the Designer

  • Deployment management metadata (commits, promotions, deployments…)

• Runtime environments (e.g. DEV, TEST, PROD)

  • Entity & package metadata as deployed to the Runtime

  • Workflow metadata

Make sure to connect to the correct environment when making Metadata Interface queries.

Connect to Metadata Interface using Swagger UI

Go to Metadata related API in Swagger UI.

Enter the tenant, installation, environment, and then press Authorize.

Enter the API keys and press Authorize after the id and the secret.

Connect to Metadata Interface using GraphQL Playground

To connect the Metadata Interface using GraphQL Playground, you need to first install the tool and then connect to Metadata Interface:

# Endpoint URL
https://external.services.saas.agiledataengine.com/external-api/api/<tenant>/<installation>/<environment>/metadata/v1/graphql

# HTTP HEADERS
{
  "X-API-KEY-ID": "<X-API-KEY-ID>" , 
  "X-API-KEY-SECRET": "<X-API-KEY-SECRET>"
}

Schema and Docs Tabs

GraphQL Playground has schema and docs tabs. The schema tab has a list of types and their fields and the docs tab has a list of queries and types related to the selected query.

Connect to Metadata Interface using Postman

To connect the Metadata Interface using Postman:

1. Install the tool and sign in.

2. Click New and GraphQL.

3. Enter the URL, replace <tenant>, <installation> and <environment> with your values (see External API):

   CODE

   https://external.services.saas.agiledataengine.com/external-api/api/<tenant>/<installation>/<environment>/metadata/v1/graphql

4. In Headers, enter the API key & API key secret:

5. In Query wait for the GraphQL introspection to load. Then you can use the schema browser to generate queries and run them against the API, for example:

Example Queries

Query schema

Use the following query to get the available types and their fields.

{__schema {types
 {name
  kind
  fields {name
  description}
  }}}

Query Entities

Query without variables

Use the following query to list 100 entities with selected fields.

# query
query{entity_list(size: 100, page: 0) 
  { page
    hasNextPage
    pageSize
    content {
      ... on ENTITY {
        id
        package_id
        schema
        name
        type
        physical_type
        source_id
      }
    }
  } 
}

Query with variables

Use the following query to list entities with selected fields using variables.

# query
query($size: Int! $page: Int!){entity_list(size: $size, page: $page) 
  { page
    hasNextPage
    pageSize
    content {
      ... on ENTITY {
        id
        package_id
        schema
        name
        type
        physical_type
        source_id
      }
    }
  } 
}


# query variables
{
 "size": 100, 
 "page": 0
 }

Query Packages

Use the following query to list 100 packages with selected fields.

# query
query($size: Int! $page: Int!){package_list(size: $size, page: $page) 
  { page
    hasNextPage
    pageSize
    content {
      ... on PACKAGE {
        id
        type
        is_deleted
        version
        name
        status
      }
    }
  } 
}

# query variables
{
 "size": 100, 
 "page": 0
}

Query Fields of Specific Entity

Query the fields of the specific entity using entity id.

# query
query($entity_id: String!){entity(id: $entity_id)
  { id
    package_id
    schema
    name
    type
    physical_type
  }
}

# query variables
{"entity_id": "986ff984-1354-4e9d-889c-34ccc1a170e9"}

Query Workflow Statuses

Use the following query to list the workflow statuses.

# query
query ($startTimeFrom: IsoDateTimeAtUtcScalar!, 
  $startTimeTo: IsoDateTimeAtUtcScalar!, $size: Int! $page: Int!) {
  workflow_stats_list(startTimeFrom: $startTimeFrom, startTimeTo: $startTimeTo, 
    size: $size, page: $page) {
    page
    pageSize
    hasNextPage
    content {
      ... on WORKFLOW_STATS {
        id
        workflow_execution_id
        name
        start_time
        end_time
        exec_time_ms
        is_successful
      }
    }
  }
}

# query variables
{
  "startTimeFrom": "2023-09-13T00:00:00.000000",
  "startTimeTo": "2023-09-13T23:59:59.999999",
  "size": 20, 
  "page": 0
}