Existing Data Apps

Overview

TNO provides a set of Data Apps that can be deployed on your Connector. Deployment is done by extending the Helm chart of your TNO Security Gateway deployment. TNO currently offers the following Data Apps:

1. Broker Data App for deploying a connector that acts as IDS Metadata Broker
2. OpenAPI Data App for exposing a backend API to an IDS based data space

In the near future, more data apps will be made open source. Including:

1. Federated Learning Data Apps for creating a Federated Learning environment within an IDS based Data Space
2. Multi Party Computation Data Apps for creating a Multi Party Computation environment within an IDS based Data Space

Metadata Broker

The Broker Data App is used to create a Metadata Broker, as specified in the International Data Spaces Reference Architecture Model. The Metadata Broker consists of a TSG Connector and a Broker Data App. The broker can be queried to discover services and other Participants in the data space. The Self Descriptions of the Connectors in the Data Space are stored in the Metadata Broker, specifically in the Fuseki triple store. The broker can be queried via SPARQL queries. The metadata broker of the TSG also comes with a User Interface , which can be used to view the Participants and Connectors that are registered in the broker. For each Connector, you can also view the Self Description and therefore also the available endpoints and resources.

Deployment

The deployment of a TSG Metadata Broker is similar to the deployment of a TSG Connector. You need a file like the one in the example below. Replace every value with # in front with your own values.

host: # Your broker URL

coreContainer:
  image: docker.nexus.dataspac.es/core-container:feature-remote-attestation-aks-amd-sev
  ingress:
      path: /(infrastructure|participants|broker)
      rewriteTarget: /$1
      clusterIssuer: letsencrypt

adminUi:
  image: docker.nexus.dataspac.es/ui/core-container-ui:feature-open-sourcing
  ingress:
    path: /ui(/|$)(.*)
    rewriteTarget: /$2
    clusterIssuer: letsencrypt

ids:
  info:
    idsid: &idsid urn:ids:connectors:Broker
    curator: urn:ids:participants:Test
    maintainer: urn:ids:participants:Test
    titles:
      - Metadata Broker
    descriptions:
      - Metadata Broker
    accessUrl:
      - "https://{{ .Values.host }}/broker"
      - "https://{{ .Values.host }}/infrastructure"
      - "https://{{ .Values.host }}/participants"
    
  daps:
    url: # URL to your DAPS

  broker:
    id: *idsid
    address: https://{{ .Values.host }}/infrastructure
    autoRegister: false

  routes:
    ingress:
      http:
        - endpoint: infrastructure
          dataApp: http://{{ template "tsg-connector.fullname" $ }}-broker-data-app-http:8080/router
        - endpoint: participants
          dataApp: http://{{ template "tsg-connector.fullname" $ }}-broker-data-app-http:8080/participants
  security:
    apiKeys:
      - id: default
        key: # Your API key here. IMPORTANT -- must start with APIKEY- so e.g. APIKEY-B2345sdl2skla23
        roles:
          - DATA_APP
          - PEF_MANAGER
    # API User Configuration
    users:
      - # -- User identifier (also used as username for the user)
        id: admin
        # -- BCrypt encoded password
        password: # your bcrypt encoded password
        # -- User role assignments
        roles:
          - ADMIN

containers:
- type: data-app
  image: docker.nexus.dataspac.es/broker/data-app:4.2.0
  name: broker-data-app
  apiKey: # Your API key here. IMPORTANT -- must start with APIKEY- so e.g. APIKEY-B2345sdl2skla23
  config:
    connectorSparqlUrl: http://fuseki-fuseki:3030/connectorData
    participantSparqlUrl: http://fuseki-fuseki:3030/participantData
    users:
      admin: # your bcrypt encoded password
- type: helper
  name: broker-ui
  image: docker.nexus.dataspac.es/broker/generic-ui:master
  services:
  - port: 80
    name: http
    ingress:
      path: /(.*)
      rewriteTarget: /$1
      clusterIssuer: letsencrypt
  environment:
  - name: BROKER_ACCESSURL
    value: http://{{ template "tsg-connector.fullname" $ }}-broker-data-app-http:8080/api/
  configMaps:
  - name: broker-ui-config
    mountPath: /app/config/
    binaryData:
      favicon.ico: # OPTIONAL: add binary for favicon here.
    data:
      # OPTIONAL: Add config here 
      config.json: |
        {}

The following commands can be used to install Fuseki and the TSG Metadata Broker:

helm upgrade --install -n NAMESPACE fuseki tsg/fuseki --version 1.0.2-master
helm upgrade --install -n NAMESPACE broker tsg/tsg-connector --version 3.2.1-master -f values.broker.yaml

OpenAPI Data App

The source code of the OpenAPI data app can be found on Gitlab.

The OpenAPI Data App is a Data App that is capable of exposing a back end API service to an IDS based Data Space using the TNO Security Gateway. By deploying the OpenAPI Data App you can easily expose any existing API service using the API’s openAPI spec.

The generic information flow of the OpenAPI data app is as follows:

As client, the backend system calls the OpenAPI Data App of its own connector. By providing the Forward-ID and Forward-Sender headers, for respectively the Agent ID of the intended receiver and the Agent ID of the sender of the message.

The endpoint of the OpenAPI Data App for forwarding the requests depends on the deployment. In general, the /openapi endpoint is available on port 8080. In case of Ingress configuration it will be something like https:///data-app/openapi. The endpoint requires as first path variable the version of the OpenAPI you want to access, e.g. 1.0.2 for the SCSN API, and this is the base URL of the request, so specific OpenAPI endpoints should be appended to this URL.

In a Sequence Diagram, this looks as follows:

In the Connector Helm chart an example is provided that configures the core container with the OpenAPI data app, which can be found in the examples folder.

Federated Learning

The source code of the Federated Learning data app can be found on Gitlab, just like the source code of the helper.

The Federated Learning Data App is a Data App that is used for orchestrating Federated Learning over IDS communication. Federated learning is a type of machine learning that preserves privacy. The developed solution is suited for Horizontal Federated Learning which uses an Algorithm to Data approach. This means that the data does not need to transfer towards an external instution, but the model travels across the different data sets and averages the results.

The data app is meant to be used in a generic way, meaning any Keras model can be uploaded to it and no changes should be made to the data apps. This means the AI researcher can focus on creating the model and does not need to spend time orchestrating their solution.

The structure of the solution is that there is one data app that is tightly integrated with the TSG, like other data apps, which handles the IDS communication. Together with this data app, a helper is deployed to handle the workload of the training itself. This helper is also coupled to the User Interface. The Data App is written in Kotlin, the helper is written in Python and uses the Tensorflow library to facilitate the use of neural networks. All models that are uploaded in the UI should be Keras models.