Core Container

Documentation

The Core Container is the heart of the connector and provides the means for secure and sovereign data sharing. Its primary focus is to ensure the identification, authentication, and authorization mechanisms are in place. It also acts as router, by leveraging Apache Camel, to route messages to other components (e.g. Data Apps) and act as gateway for exchanging data with other components in a data space.

Overview

Core Container - Click to enlarge

Core Container

The Core Container is built around the Spring Boot framework and Apache Camel. And is devided in subsystems that each provide a part of the functionality of the core container.

The Core Container is written in Kotlin, which combines Java-based object-oriented programming and functional programming, which runs on a Java Virtual Machine (JVM). The build process is configured using Gradle and the build artifact is designed to be a Docker image. For the creation of a Docker image, the JIB Gradle plugin is used that allows the building of a Docker image without requiring a Docker daemon to be present. This fact makes that the build process can be integrated into continous integration (CI) pipelines easily.

Ensure security of docker images

Before deploying a version of the core-container, you can scan the image using Trivy to obtain a CIS benchmark. This CIS benchmark gives you an overview of the risks when you deploy the connector. An old image with old dependencies might include severe bugs that have not been patched yet. Because our connector is open source, you can always edit the dependency and verify the tests still succeed. If you want to report a security flaw, we point you to the README of the Core Container for a set of instructions on how to bring this to the attention of the main developers of the core container.

To verify the image of the core container install trivy and follow the steps in the README. Specifically the following command:

trivy image docker.nexus.dataspac.es/core-container:<version>

Subsystem overview

The Core Container repository is composed of several subsystems responsible for parts of the core container:

camel-daps-processor

This module contains Apache Camel processors to intercept ingress and egress routes.

The Camel Processors are used in all the Camel routes that are generated, primarily in the ids-resource-manager module.

DAPS Token Injector

The DapsTokenInjector class is a Camel Processor and Spring Component that can be automatically detected and used in a Spring Context.

The DAPS Token Injector will ensure that processed egress messages always contain an IDS Dynamic Attribute Token (DAT) received from the Dynamic Attribute Provisioning Service (DAPS). Acquiring tokens is handled by ids-token-manager module

DAPS Token Verifier

The DapsTokenVerifier class is a Camel Processor and Spring Component that can be automatically detected and used in a Spring Context.

The DAPS Token Verifier will ensure that processed ingress messages always contain a valid IDS Dynamic Attribute Token (DAT) from a trusted DAPS. Verification of claims made in tokens can be configured by means of the ids.daps.verification property:

  • IDSA (default): Verification conforming to the DAPS v2 specification (see IDS-G - Components - Identity Provider - DAPS). Will verify the ids.daps.requiredSecurityProfile property to validate the required security profile is met.
  • NONE: Disable verification of claims in the token, only ensures the token is signed by a trusted DAPS

camel-multipart-processor

This module contains Apache Camel processors to process ingress and egress HTTP Mime Multipart messages.

The Camel Processors are used in all the Camel routes that are generated, primarily in the ids-resource-manager module.

Multipart Input Processor

The MultiPartInputProcessor class is a Camel Processor and Spring Component that can be automatically detected and used in a Spring Context.

The Multipart Input Processor will transform an incoming HTTP Mime Multipart request towards a header based approach used in Apache Camel. The multipart formatted request will contain the following parts:

  • header: Required de.fraunhofer.iais.eis.Message (in JSON-LD representation) metadata of the request following the IDS Information Model.
  • payload (optional): Arbitrary payload of the request.

The header part will be stored in the IDS-Header header in the Camel Exchange, the payload part will be stored in the body of the Camel Exchange.

Multipart Output Processor

The MultiPartOutputProcessor class is a Camel Processor and Spring Component that can be automatically detected and used in a Spring Context.

The Multipart Output Processor will inverse the Multipart Input Processor and will construct a HTTP Mime Multipart message with header and payload parts. For this it will use the IDS-Header header and the exchange payload. There are no verifications enabled for the output processor, so the header and payload will be transparently passed through and only the formatting will be changed.

ids-artifact-manager

The IDS Artifact Manager module is a self-contained module to support simple data exchanges without additional Data Apps.

Encryption at rest

Using the standard configuration, the artifacts uploaded to the TSG are encrypted at rest. This is done using the Advanced Encryption Standard (AES) with a key of 128 bits. Encryption of files stored in the core container protects sensitive data from unauthorized access and breaches. The artifacts are encrypted with a configurable AES encryption key.

For data at rest AES/GCM/NoPadding is used, this refers to specific encryption configuration using the Advanced Encryption Standard (AES) algorithm in Galois/Counter Mode (GCM) mode without padding. The initialisation vector used for GCM is stored alongside the encrypted data in a iv file.

Spring REST Controllers

The Artifact Manager contains 2 internal API controllers for administration of artifacts and for requesting artifacts.

Consumer

The Consumer controller offers the possibility to request artifacts from other IDS connectors. Also, the consumer controller is capable of starting the Contract Negotiation process, based on a predefined ContractOffer (e.g. stored next to the artifact metadata in the Broker)

Provider

The Provider controller offers the possibility to manage artifacts published by the connector, by means of CRUD requests.

Next to the actual artifact, a ContractOffer can be provided to limit the access rights to the artifact. This ContractOffer will be scoped to reflect the uploaded artifact, by setting the target property of all rules in the ContractOffer to the identifier of the artifact.

Artifact exchange process

The sequence diagram below shows the generic process for the interaction of users with the Artifact Manager. Containing the processes to upload a new artifact, perform the contract negotiation, and downloading the artifact.

sequenceDiagram actor C as Consumer participant CC as Consumer Connector participant PC as Provider Connector actor P as Provider P->>PC: /api/artifacts/provider
With optional ContractOffer activate P activate PC PC->>PC: Store artifact PC-->>P: deactivate P deactivate PC C->>CC: /api/artifacts/consumer/contractRequest activate C activate CC CC->>PC: ContractRequestMessage activate PC alt Contract Request accepted PC-->>CC: ContractAgreementMessage CC-->>C: ContractAgreement else Contract Request denied PC-->>CC: ContractRejectionMessage CC-->>C: ContractRejection end C->>CC: /api/artifacts/consumer/artifact CC->>PC: ArtifactRequestMessage PC-->>CC: ArtifactResponseMessage deactivate PC CC-->>C: Artifact deactivate C deactivate CC

ids-clearing

The IDS clearing module is responsible for clearing ingress and egress IDS messages.

Currently, the clearing is done in-memory combined with logging to standard out, with the possibility to retrieve the clearing logs via an API endpoint.

The logs are linked with signatures of the previous statement in order to prevent removal or additions to the entries after the fact.

The logs that are currently included:

  • Access control decisions from the Policy Enforcement Framework, both granted and rejected requests. In the following format: timestamp=TIMESTAMP; Access Control: consumer=CONSUMER_ID, provider=PROVIDER_ID, senderAgent=SENDER_AGENT, recipientAgent=RECIPIENT_AGENT, contractId=CONTRACT_ID, decision=DECISION
  • Data requests, both granted and rejected requests. In the following format:
    timestamp=TIMESTAMP; Data Access: consumer=CONSUMER_ID, provider=PROVIDER_ID, senderAgent=SENDER_AGENT, recipientAgent=RECIPIENT_AGENT, contractId=CONTRACT_ID, identifier=RESOURCE_IDENTIFIER
  • Backup/restore and configuration changes. In the following format:
    timestamp=TIMESTAMP; Configuration Interaction: responsible=USER_ID, category=CATEGORY, operation=OPERATION:CHANGED_CONTENT, status=CONFIG_HTTP_STATUS
  • Exchanged IDS messages, both ingress as egress. In a JSON format retrievable via the administrative API.

Where all timestamps within the logging are formatted using UNIX timestamps and within the JSON formats retrievable from the administrative API the ISO 8601 format is used.

In case there are errors during the validation of signatures exceptions will be thrown that will kill the connector. With either an InvalidKeyException or a SignatureException.

ids-configuration

Helper module that contains generic configuration of the core container and common Spring Beans used throughout the repository.

The module contains the following packages:

  • events: Spring ApplicationEvent support, to indicate modifications that impact the Self Description of the Connector.
  • http: HttpClient provider, contains the Truststore configuration so that servers that use their IDS certificate for TLS encryption are trusted.
  • infomodel: Helper functions and message converters for IDS Information Model related classes.
  • keystore: Parser for PEM-based certificates and private keys
  • model: Main Configuration model used for Spring Configuration Properties that are loaded via application.yaml, application.properties, or environment variables
  • multipart: MultiPartMessage (de)serialization support.
  • reloading: Hot-reloading of Spring properties and classes support. (Note: does not fully work)
  • serialization: Provides SerializationHelper as Spring Beans
  • validation: Adds additional validation constraints that can be used by model classes

ids-main

This module collects all of the subsystems and provides the main class for execution.

This module also contains some configuration and beans that require other modules to work:

  • endpoint: Apache Camel processor that adds Camel HTTP components parameters to forward addresses
  • errorhandling: Apache Camel processor that will be used when an exception is thrown in one of the exchange routes.
  • healthcheck: Health check endpoint for orchestration (e.g. Kubernetes) purposes
  • idscp: IDSCPv2 configuration, used to configure de.fhg.aisec.ids:camel-idscp2
  • sslcontextparameters: SSLContextParameters provider for usage in apache Camel routes, based on the Truststore configuration

Google Container Tools Jib Docker builds

This module contains the configuration for the Docker image build process by using the Jib library. This is located in the build.gradle.kts file.

For local builds, the jibDockerBuild task can be used to build the image using Docker on the same machine (e.g. ./gradlew jibDockerBuild (in the root of the repository)). For other builds the jib task can be used, that automatically uploads the Docker image to the registry (requires REGISTRY_USERNAME & REGISTRY_PASSWORD environment variables, or being logged in to the registry via the Docker CLI).

ids-orchestration-manager

This module contains the Orchestration Manager for orchestrating containers alongside the core container (e.g. Data Apps).

Currently supported container environments:

  • Kubernetes

This module is actively used by the ids-workflow-manager module to start, stop, and manage containers for workflows.

ids-pef

This module contains the embedded Policy Enforcement Framework.

The Policy Enforcement Framework follows the XACML architecture:

Policy Administration Point

The Policy Administration Point (PAP) offers a management API for CRUD operations for ContractOffers and ContractAgreements in the administration of policies.

The ContractOffers are used in the negotiation process to be able to automatically negotiate ContractAgreements. The ContractAgreements are retrieved by the PDP at the moment a decision has to be made whether or not to allow requests.

Policy Enforcement Point

The Policy Enforcement Point (PEP) makes sure that policies are enforced for all of the messages that flow through the PEP. The PEP is an Apache Camel processor that can be included in Camel routes, which makes that the messages in that route will be enforced.

Policy Decision Point

The Policy Decision Point (PDP) makes the actual decision whether or not to allow a request, based on the context of the request. This includes the policy itself (from the PAP), environmental properties (from the PEP), additional attributes (from the PIP).

Policy Information Point

The Policy Information Point (PIP) provides additional information that might be required for taking a decision. At this moment, just the times a contract/resource are used is stored in the PIP.

Negotiation

The Negotiation Processor will actively intercept messages related to the negotiation process (ContractRequestMessage, ContractOfferMessage, ContractAgreementMessage, and ContractRejectionMessage) and handle the automated negotation process.

Note: Currently automated negotiation only supports ContractRequests that are similar or of narrower scope than the related ContractOffer

ids-resource-manager

This IDS Resource Manager module provides the capability to manage the IDS metadata of resources the connector offers in the network.

The ResourceManagerService is used by the ids-selfdescription module as source of the resourceCatalog property in the Connectors self-description that will be published at the broker.

Data Apps are able to use the endpoints provided by the ResourceController to manage the resources that the Data App offers.

ids-route-manager

The IDS Route Manager module manages all of the dynamic Camel routes in the core container.

The routes originate either from the Spring Configuration properties (ids.routes), or from the ids-workflow-manager module.

The CamelRoutesGenerator generates XML-based Apache Camel routes from Route objects from the configuration model (ids-configuration module).

The CamelRouteManager has support for both the XML represented routes as the Java-DSL represented routes (in the form of a RouteBuilder object)

ids-security

The IDS Security module is responsible for the security of non-IDS communication. Which includes:

  • the management API
  • the Data App → Core Container API

Authentication for these resources is done by either API key authentication or by User credential authentication. For User credential authentication, a token is requested at the /auth/signin which can be used for requests to the API.

Security Roles

Access to resources of the API is role-based, the following roles are currently supported:

  • ADMIN: Administrator role for the primary administrator that has full access and inherits all of the other roles
  • DATA_APP: Data App role that can be given to Data Apps for managing the resources that they offer
  • READER: Reader role that is allowed to read all resources in the core container but not modify them
  • ARTIFACT_PROVIDER_MANAGER: Artifact Provider Manager role that is allowed to administrate provided artifact
  • ARTIFACT_PROVIDER_READER: Artifact Provider Reader role that is allowed to list provided artifacts
  • ARTIFACT_CONSUMER: Artifact Consumer role that is allowed to request artifacts from other connectors in the network
  • ORCHESTRATION_MANAGER: Orchestration Manager role that is allowed to orchestrate containers
  • ORCHESTRATION_READER: Orchestration Reader role that is allowed to list orchestrated containers
  • PEF_MANAGER: Policy Enforcement Manager role that is allowed to initiate new contracts and contract negotiations
  • PEF_READER: Policy Enforcement Reader role that is allowed to list agreed upon contracts and contract offers
  • RESOURCE_MANAGER: Resource Manager role that is allowed to modify the resources provided by the core container and data apps
  • RESOURCE_READER: Resource Reader role that is allowed to list resources provided by the core container and data apps
  • ROUTE_MANAGER: Route Manager role that is allowed to modify Camel routes offered by the Core Container
  • ROUTE_READER: Route Reader role that is allowed to list Camel routes offered by the Core Container
  • DESCRIPTION_READER: Description Reader role that is allowed to send Description Request messages to other connectors in the network
  • WORKFLOW_MANAGER: Workflow Manager role that is allowed to initiate new workflows
  • WORKFLOW_READER: Workflow Reader role that is allowed to list workflows and show the results and status of the workflow

Controller Configuration

Spring controllers can limit the access to endpoints by providing the @Secured annotation, which requires a list of roles that have access to the endpoints. This can be done both on class-level (indicating that for each of the methods these roles are required) or on method-level, these options are combined so you’d either need access on class-level or on method-level.

The SecurityRoles class provides constant values that can be used in the annotations:

@RestController
@Secured(SecurityRoles.ARTIFACT_PROVIDER_MANAGER)
class Controller

@GetMapping
@Secured(SecurityRoles.ARTIFACT_PROVIDER_READER)
fun get()

ids-selfdescription

The IDS Self Description module is responsible for constructing the self description of the connector and interact with the Broker to publish the self description.

Self Description Generator

The SelfDescriptionGenerator component constructs the self description metadata. The sources used for this are:

  • Spring Configuration properties: The information in the ids.info.* properties is used to construct the generic information about the connector and who is owner of the connector.
  • Resource Manager Service: The complete resource catalog exposed by the connector is managed by the ids-resource-manager module

When receiving the SelfDescriptionReloadEvent, the SelfDescriptionGenerator will generate the up-to-date self description. After the self description is generated, the SelfDescriptionChangeEvent is triggered.

Broker Registration Service

The BrokerRegistrationService will publish the connector’s metadata at the configured broker. For this to work, the ids.broker.autoRegister property should be true.

Note: Manual registration of the connector is not yet supported

The BrokerRegistrationService will try the initial Broker registration in a retryable fashion (configurable via ids.broker.registrationMaxRetries & ids.broker.registrationBackoffPeriod).

To make sure the self description is updated and the Broker knows the connector is still available, the BrokerRegistrationService will periodically publish its self description (configurable via broker.brokerHealthCheckInterval).

Description Processor

The DescriptionProcessor provides the functionality of handling incoming DescriptionRequestMessages. By either providing the full self description, or a specific portion if the requestedElement property is filled in the request message. The processor is used in the Camel routes for the self description.

Description Controller

The DescriptionController provides an API to send a DescriptionRequest to another connector. Primary used in combination with the ArtifactConsumerController from the ids-artifact-manager module, for retrieving ContractOffers provided with resources in the self description.

ids-token-manager

The IDS Token Manager module is responsible for the interaction with Dynamic Attribute Provisioning Service (DAPS) instances.

The primary exposed functionality is:

  • Requesting a Dynamic Attribute Token (DAT) from the DAPS for this connector
  • Verifying a Dynamic Attribute Token (DAT) from an incoming message. Note: only the signature of the token will be validated. The validity of the claims is verified in the camel-daps-processor module

Based on the configuration, the relevant ClientAssertionGenerator and TokenManagerService implementations will be loaded.

The ClientAssertionGenerator will make sure the right identifier is used by recomputing the identifier directly from the certificate. The used format for this identifier is:

{SubjectKeyIdentifier}:keyid:{AuthorityKeyIdentifier}

Where SubjectKeyIdentifier is the ASN1 object from the certificate with id 2.5.29.14.
And AuthorityKeyIdentifier is the ASN1 object from the certificate with id 2.5.29.35.

ids-workflow-manager

The IDS Workflow Manager module provides the functionality to manage workflows, both internal workflows (without communication with other connectors) as external workflows (with communication with other connectors) are supported.

Build process

The following steps can be executed to build the Core Container and automatically create a Docker image for it.

To build execute (automatically pushes the image to the repository):

./gradlew clean assemble ciTest jib

The default image name structure that is used is:

docker.nexus.dataspac.es/core-container:{GIT_BRANCH_NAME}
docker.nexus.dataspac.es/core-container:{GIT_BRANCH_NAME}-{GIT_COMMIT_HASH}

To build the image for local usage (requires a Docker daemon to be reachable):

./gradlew clean assemble ciTest jibDockerBuild

To execute the integration and local tests (requires a Docker daemon to be reachable):

./gradlew clean assemble test

NOTE: The Spotless plugin is used to automatically set the copyright headers and formatting of files In case of errors during Spotless Check tasks, run the following task to automatically fix the errors: ./gradlew spotlessApply

Didn't find what you were looking for?