API

The Core Container uses and exposes several APIs for interaction with the Core Container as well as for Connector-to-Connector and Data App-to-Connector communication.

The following default ports are used to expose the different APIs:

• Port 8080: Inter Connector API - HTTP MIME Multipart. Secured by requiring Dynamic Attribute Tokens (DATs).
• Port 8081: Metrics API. Not secured, intended to only be exposed within a cluster.
• Port 8082: Admin API. Secured with user credentials and API keys.

Inter Connector API

The Inter Connector API is based around the Information Model Messages and follows this basic structure:

• header: An ids:Message subclass.
• payload: Optional arbitrary payload, might be scoped given the header class.

IDS allows three communication protocols: HTTP MIME Multipart, IDSCPv2, IDS REST.

In cases where the connector can’t provide a suitable response, a RejectionMessage (or ContractRejectionMessage in case of a contract negotiation) is returned with a limited set of information to prevent attacks based on the response of the connector. The logs of the connector often provide a more detailed explanation that can be used for root-cause analysis. All requests based on the IDS Information Model are validated against the SHACL shapes defined in the Information Model. Errors in the messages always result in a RejectionMessage.

HTTP MIME Multipart

The HTTP MIME Multipart protocol follows the RFC1341 7.2 standard, with either mixed (multipart/mixed) or form-data (multipart/form-data). This is the default protocol used by the Core Container.

The HTTP messages are composed out of a header part with a JSON-LD represenstation of the ids:Message subclasses and optional payload part.

An example of an HTTP MIME Multipart message containing a ConnectorUpdateMessage as header together with a self-description as payload:

POST /router HTTP/1.1
Host: localhost:8080
Content-Type: multipart/form-data; boundary=VWo5W0Rmj8P8IHyC84CxRYpQNM62IP
Accept: */*

--VWo5W0Rmj8P8IHyC84CxRYpQNM62IP
Content-Disposition: form-data; name="header"
Content-Type: application/ld+json

{
  "@context" : {
    "ids" : "https://w3id.org/idsa/core/",
    "idsc" : "https://w3id.org/idsa/code/"
  },
  "@type" : "ids:ConnectorUpdateMessage",
  "@id" : "https://w3id.org/idsa/autogen/connectorUpdateMessage/5d84ce59-0b34-4483-81f5-7126976fa482",
  "ids:modelVersion" : "4.1.0",
  "ids:issued" : {
    "@value" : "2021-11-19T12:55:50.046+01:00",
    "@type" : "http://www.w3.org/2001/XMLSchema#dateTimeStamp"
  },
  "ids:issuerConnector" : {
    "@id" : "urn:ids:test"
  },
  "ids:recipientConnector" : [ {
    "@id" : "urn:ids:broker"
  } ],
  "ids:senderAgent" : {
    "@id" : "urn:ids:test"
  },
  "ids:recipientAgent" : [ {
    "@id" : "urn:ids:broker"
  } ],
  "ids:securityToken" : {
    "@type" : "ids:DynamicAttributeToken",
    "@id" : "https://w3id.org/idsa/autogen/dynamicAttributeToken/1368e56a-5f13-44fb-95d3-c9577c96a134",
    "ids:tokenValue" : "DUMMY",
    "ids:tokenFormat" : {
      "@id" : "https://w3id.org/idsa/code/JWT"
    }
  },
  "ids:affectedConnector" : {
    "@id" : "urn:ids:test"
  }
}
--VWo5W0Rmj8P8IHyC84CxRYpQNM62IP
Content-Disposition: form-data; name="payload"
Content-Type: application/ld+json

{
  "@context" : {
    "ids" : "https://w3id.org/idsa/core/",
    "idsc" : "https://w3id.org/idsa/code/"
  },
  "@type" : "ids:TrustedConnector",
  "@id" : "urn:ids:test",
  "ids:maintainer" : {
    "@id" : "urn:ids:test"
  },
  "ids:resourceCatalog" : [ ],
  "ids:description" : [ {
    "@value" : "Test",
    "@type" : "http://www.w3.org/2001/XMLSchema#string"
  } ],
  "ids:curator" : {
    "@id" : "urn:ids:test"
  },
  "ids:inboundModelVersion" : [ "4.1.0", "4.1.1" ],
  "ids:outboundModelVersion" : "4.1.0",
  "ids:hasAgent" : [ ],
  "ids:securityProfile" : {
    "@id" : "https://w3id.org/idsa/code/TRUST_SECURITY_PROFILE"
  },
  "ids:extendedGuarantee" : [ ],
  "ids:hasDefaultEndpoint" : {
    "@type" : "ids:ConnectorEndpoint",
    "@id" : "https://w3id.org/idsa/autogen/connectorEndpoint/bfcb03d5-8724-4c7b-af16-7b4174a6e419",
    "ids:endpointInformation" : [ ],
    "ids:endpointDocumentation" : [ ],
    "ids:accessURL" : {
      "@id" : "https://test"
    }
  },
  "ids:hasEndpoint" : [ {
    "@type" : "ids:ConnectorEndpoint",
    "@id" : "https://w3id.org/idsa/autogen/connectorEndpoint/bfcb03d5-8724-4c7b-af16-7b4174a6e419",
    "ids:endpointInformation" : [ ],
    "ids:endpointDocumentation" : [ ],
    "ids:accessURL" : {
      "@id" : "https://test"
    }
  } ],
  "ids:title" : [ {
    "@value" : "Test",
    "@type" : "http://www.w3.org/2001/XMLSchema#string"
  } ]
}
--VWo5W0Rmj8P8IHyC84CxRYpQNM62IP--

By default, the following HTTP MIME Multipart endpoints are configured for the connector:

• /selfdescription: Exposes the self description of the connector. With by default support for POST messages containing an ids:DescriptionRequestMessage message for requesting either the complete self description or only a sub-part of it. These messages must be sent from another connector with a valid DAT. Support for simple GET requests can be configured, which results in non-restricted access to the self description (without any means of requesting specific parts of the self description).
• /router/artifacts: Exposes the endpoint for artifact request handling, with support for: ids:ArtifactRequestMessages, ids:DescriptionRequestMessages, and contract negotiation related messages.

IDSCPv2

The IDS Communication Protocol version 2 (IDSCPv2) is a custom TLS-based stateful protocol. It consists of a transport layer protocol responsible for setting up a mutual authenticated, encrypted and integrity protected communication channel, and an application layer protocol that allows for the actual data exchange to be exchanged over the session.

The application layer uses Protobuf for the serialization of the header and payload parts. The message definition is as follows:

syntax = "proto3";
message IdsMessage {
  // Arbitrary header string
  string header = 1;
  // The actual, generic message payload
  bytes payload = 2;
}

The header part should be represented in JSON-LD format and the payload part can contain an arbitrary byte array.

IDS REST

Disclaimer: The Core Container does not support IDS REST at this moment.

The IDS REST protocol is intended to be the replacement of the HTTP MIME Multipart protocol, by following more closely the Linked Data Platform W3C recommendation.

The IDS REST protocol deviates from the Multipart and IDSCPv2 protocols with respect to the representation of the header part, since it doesn’t use the JSON-LD representation for the header. Instead, a combination of HTTP method, HTTP URL, and HTTP headers is used to represent the header.

The protocol is not finalized and published yet, as soon as it is it will be linked here. This also is the reason the TSG components do not support IDS REST at this moment.

Data App API

The current implementations of Data Apps use the HTTP MIME Multipart protocol for exchanging information with the Core Container. At this moment, this is not standardized yet and might be subject to change in the future.

Next to the standard IDS communication, a Data App can also use the Admin API to manipulate the Core Container. For instance, to provide resource descriptions to the Resource Manager module using the /api/resources endpoints.

Both the Multipart and the Admin API endpoints can be secured with API keys, as is detailed below in Security.

Metrics API

Metrics are exposed via the Spring Boot Actuator that allows to expose information that might be required for production-ready deployments of the core container. By default the metrics are exposed on a seperate port (8081) to allow for easily exposing the metrics only within a Kubernetes cluster and not to the outside world. Also, the actuator endpoints are not secured with any credentials, to allow for easy configuration within Kubernetes, Prometheus, and any other tooling that might use the actuators for monitoring and orchestration.

The base URL of the actuators is: http://localhost:8081/actuator

The actuators that are configured are:

• /actuator/health: Simple health check to check whether Spring is successfully started and running
• /actuator/prometheus: Metric endpoint formatted to be used directly within Prometheus
• /actuator/metrics: Metric endpoint for extracting single metrics (see: Spring Boot Docs)

The available metrics are generic process & JVM metrics, combined with the following Camel metrics (per route):

• CamelExchangesTotal: Counter of total exchanges (i.e. messages) per Camel route
• CamelExchangesFailed: Counter of failed exchanges (i.e. messages) per Camel route
• CamelExchangesFailuresHandled: Counter of failed exchanges that are handled (i.e. messages) per Camel route
• CamelExchangesSucceeded: Counter of succeeded exchanges (i.e. messages) per Camel route
• CamelRoutePolicy: Processing time per Camel route
• CamelMessageHistory: Processing time per node per Camel route

Admin API

The internal API of the Core Container is meant to be used for administration of the Core Container by either users or deployed Data Apps. It follows largely the structure of the modules of the Core Container.

The API is used by the Web User Interface that allows users to interact with the Core Container. As well as, by Data Apps that are able to communicate with API keys with the endpoints of the API. See the Security section for more information on the security of the API.

All inputs towards the administrative API are validated against Kotlin Data Classes that ensure that there is no malformed input allowed, this includes the messages returned by the API.

Artifact Management

The Artifact Management API is used to interact with the built-in Artifact handling of the Core Container.

Clearing controller

The Clearing controller API is used to retrieve information on cleared messages.

Orchestration Management

The Orchestration Management API is used to interact with the orchestration manager that allows for orchestration of data apps and helper containers.

Policy Enforcement Management

The Policy Enforcement Management API is used to interact with the built-in Policy Enforcement Framework. More specifically, the API is used to interact with the Policy Administration Point to manage agreed upon contracts and contract offers.

Resource Management

The Resource Management API is used to interact with the Resource Management of the Core Container, which is used to generate the self-description of the Connector. These endpoints are primarily used by Data Apps that provide resources for the Connector.

Route Management

The Route Management API is used to interact with the Camel Route Manager to administrate Camel routes and view metrics of the routes.

Authentication Management

The Authentication Management API is used to interact with the Authentication Manager to administrate users and API keys that have access to the Admin API.

Self Description

The Self Description API is used to request metadata from other connectors or from the Broker.

Workflow Management

The Workflow Management API is used to interact with the Workflow Manager.

Security

Access to the public endpoints of the connector are handled via the IDS protcols, but access to the internal endpoints (both Admin API and Data App API) can be secured via a token-based approach. At this moment, the security feature is disabled by default (security.enabled property), which allows any request to be handled without authentication. When this feature is enabled, there is an option to either authenticate with user credentials or with API keys.

For user-based authentication, a user must request a token from the Core Container at the /auth/signin endpoint with its credentials (userid and password). When successfully authenticated, the user receives a JSON Web Token (JWT) that it must use in further requests to the Admin API as Bearer Authentication HTTP header. Tokens of users are valid for 1 hour.

A user can provide 10 wrong credentials within 15 minutes before the user account is locked for 15 minutes. No detailed information is shown in the error messages indicating that the user account is locked, as well as, no additional information is provided what might be wrong with the request. All failed attempts will be containing non-descriptive message like “Invalid username/password supplied”.

For API keys, the configured API key can be used directly in the Bearer Authentication HTTP header and is primarily intended for Data Apps or applications interacting with the Core Container. API keys are valid until they are deleted via the Admin API.

Each user and API key can be assigned to roles to only allow access to specific API endpoints. The list of valid roles is:

Especially for API keys, it is wise to limit the scope of the key to just the API endpoints a Data App will use, since API keys do not have an expiration date which increases the impact of leaked API keys.

Passwords for users are encrypted using BCrypt, in both the configuration files and via the API passwords must be presented in BCrypt encoded form. It is required to use a cost factor of 12 or higher when hashing the password with the BCrypt hash function. BCrypt cost factors result in a number of hashing round equal to 2 ^ COST_FACTOR, for a cost factor of 12 this results in 4096 hashing rounds. Next to this, there is an option to perform password rotation. With the password rotation, it is not possible to verify whether two passwords are the same because of the bcrypt hashing. In the UI password strength is enforced via the forms, where a password policy of: at least 12 characters, at least 1 uppercase character, at least 1 lowercase character, at least one special character. In the UI also a requirement for API keys is enforced, which ensures all API keys are at least 20 characters long (not including APIKEY-). Both minimum required lengths can be configured by applying environment variables to the UI, for the password length the variable PASSWORD_LENGTH is used and for the API key length APIKEY_LENGTH is used. If these variables are set lower than the default values of respectively 12 and 20, the default values are used instead.