Configuration Reference

This document provides a comprehensive reference for all configuration files used in the Integration Platform.

Configuration Structure Overview

All configuration files are stored in Google Cloud Storage bucket: {project}-integration-service

gs://{project}-integration-service/
├── message_config.json              # Message type definitions
├── schedules/schedule_config.json   # Scheduling and routing configuration
├── connections/                     # Connection configurations
│   ├── {connection_id}.json
│   └── ...
├── field_map_configs/              # SFTP field mapping configurations
│   ├── {message_type}.json
│   └── ...
├── file_tags/                      # File tag configurations
│   ├── {message_type}.json
│   └── ...
├── maps/                           # SOAP field mapping configurations
│   ├── {message_type}.json
│   └── ...
├── rules/                          # Validation rule configurations
│   ├── {message_type}_{process_type}.json
│   └── ...
└── schemas/                        # XSD schemas
    ├── {message_type}.xsd
    └── ...

---

message_config.json

Defines all supported message types for SOAP, SFTP and http-pull integrations.

Structure

{
  "{MessageType}": {
    "mmchub_inbound_type": boolean,
    "sftp_inbound_type": boolean,
    "http_pull_inbound_type": boolean,
    "xsd_schemas": {
      "code_list": [string],
      "folder": string,
      "file": string
    },
    "message_maps": {
      "folder": string,
      "file": string
    },
    "rule_configs": [
      {
        "process_type_id": string,
        "folder": string,
        "file": string
      }
    ]
  }
}

Field Reference

Field	Type	Required	Default	Description
mmchub_inbound_type	boolean	Yes	-	Enable for SOAP MMC-Hub inbound messages
sftp_inbound_type	boolean	Yes	-	Enable for SFTP inbound messages
http_pull_inbound_type	boolean	No	false	Enable for http-pull (HTTP download) inbound messages
xsd_schemas	object	No		XSD schema configuration for validation
xsd_schemas.code_list	array	No	[]	List of XSD codelist files
xsd_schemas.folder	string	No	-	Folder containing schema files
xsd_schemas.file	string	No	-	Main XSD schema filename
message_maps	object	No		SOAP field mapping configuration
message_maps.folder	string	No	-	Folder containing mapping files
message_maps.file	string	No	-	Mapping configuration filename
rule_configs	array	No	[]	Validation rule configurations
rule_configs[].process_type_id	string	Yes	-	Unique process type identifier
rule_configs[].folder	string	Yes	-	Folder containing rule files
rule_configs[].file	string	Yes	-	Rule configuration filename

Examples

SOAP Message with Validation:

{
  "GL_MarketDocument": {
    "mmchub_inbound_type": true,
    "sftp_inbound_type": false,
    "xsd_schemas": {
      "code_list": ["urn-entsoe-eu-wgedi-codelists.xsd"],
      "folder": "schemas",
      "file": "GL_MarketDocument.xsd"
    },
    "message_maps": {
      "folder": "maps",
      "file": "MarketDocument.json"
    },
    "rule_configs": [
      {
        "process_type_id": "A26",
        "folder": "rules",
        "file": "GL-MarketDocument_A26.json"
      }
    ]
  }
}

Simple SFTP Message:

{
  "DRE": {
    "sftp_inbound_type": true,
    "xsd_schemas": {},
    "message_maps": {},
    "rule_configs": []
  }
}

---

schedules/schedule_config.json

Defines scheduling configuration for SOAP, SFTP and http-pull message retrieval.

Structure

{
  "schedule_types": {
    "{schedule_name}": {
      "type": string,
      "description": string,
      "interval": string,
      "expression": string,
      "timezone": string
    }
  },
  "schedules": [
    {
      "name": string,
      "enabled": boolean,
      "type": string,
      "connection_id": string,
      "schedule": string,
      "namespaces": [
        {
          "namespace": string,
          "reroute_namespace": string
        }
      ],
      "description": string
    }
  ]
}

Schedule Types Field Reference

Field	Type	Required	Default	Description
type	string	Yes	-	"interval", "cron", "manual", or "disabled"
description	string	No	""	Human-readable description
interval	string	Conditional	-	Interval as minutes or hours, e.g. "1m", "15m", "1h" (if type="interval"). Intervals are counted from midnight, so a "24h" interval always runs at 00:00
expression	string	Conditional	-	Five-field cron expression, e.g. "30 0 * * *" for 00:30 every day (if type="cron")
timezone	string	No	"UTC"	IANA timezone a cron expression is evaluated in, e.g. "Europe/Paris" (if type="cron"). Interval schedules ignore it

A cron schedule follows daylight saving time in its timezone: 0 6 * * * with Europe/Paris runs at 06:00 Paris time all year. When the clocks go forward, a time that falls in the skipped hour runs once right after the change; when they go back, a time that occurs twice runs only the first time. Expressions resolve to the minute and only fire on minutes the scheduler runs: every minute by default, but scheduler_schedule can change that per environment (for example */30 * * * *, where 15 * * * * would never fire). An invalid expression or timezone is logged and the schedule does not run. Use UTC for schedules that run more than once an hour: in a timezone with daylight saving time, the runs in the repeated hour are skipped once a year.

Schedules Field Reference

Field	Type	Required	Default	Description
name	string	Yes	-	Unique schedule identifier
enabled	boolean	Yes	-	Enable/disable the schedule
type	string	Yes	-	"soap", "sftp" or "http_pull"
connection_id	string	Yes	-	Reference to connection config
schedule	string	Yes	-	Reference to schedule_type name
namespaces	array	Yes	-	Namespace entries to process (see below)
description	string	No	""	Human-readable description

Namespace Entry Field Reference

Each entry in namespaces is an object (a legacy plain namespace-id string is still accepted and treated as an entry without a fallback):

Field	Type	Required	Default	Description
namespace	string	Yes	-	Namespace ID to poll
reroute_namespace	string	No	the polled namespace	Namespace a pulled message is rerouted to when its datasource does not resolve in the polled namespace (replaces the former FALLBACK_EWX_NS service setting)

Examples

{
  "schedule_types": {
    "every_minute": {
      "type": "interval",
      "description": "Run every minute",
      "interval": "1m",
      "timezone": "UTC"
    },
    "every_5_minutes": {
      "type": "interval",
      "description": "Run every 5 minutes",
      "interval": "5m",
      "timezone": "UTC"
    },
    "daily_0030": {
      "type": "cron",
      "description": "Run daily at 00:30 UTC",
      "expression": "30 0 * * *",
      "timezone": "UTC"
    }
  },
  "schedules": [
    {
      "name": "soap_hub_schedule",
      "enabled": true,
      "type": "soap",
      "connection_id": "soap_hub_primary",
      "schedule": "every_minute",
      "namespaces": [
        {"namespace": "enrx_org_004"},
        {"namespace": "enrx_org_006", "reroute_namespace": "enrx_org_004"}
      ],
      "description": "Check the SOAP hub for new messages"
    },
    {
      "name": "acme_sftp_schedule",
      "enabled": true,
      "type": "sftp",
      "connection_id": "acme_sftp",
      "schedule": "every_5_minutes",
      "namespaces": [{"namespace": "enrx_org_004"}],
      "description": "Process DRE files from Acme SFTP"
    },
    {
      "name": "cre_publication_cr_daily",
      "enabled": true,
      "type": "http_pull",
      "connection_id": "cre_publication_cr",
      "schedule": "daily_0030",
      "namespaces": [{"namespace": "enrx_org_001"}],
      "description": "Download the CRE publication daily at 00:30 UTC"
    }
  ]
}

---

connections/{connection_id}.json

Defines SFTP, SOAP or http-pull connection configurations. A connection can be switched off with "enabled": false — a disabled connection is not polled by the inbound scheduler, and outbound messages for its namespaces are processed and audited but not sent (this replaces the former SKIP_SENDING service setting).

SOAP (MMC-Hub) Connection Structure

{
  "enabled": boolean,
  "party_id": string,
  "description": string
}

SOAP Field Reference

Field	Type	Required	Default	Description
enabled	boolean	No	true	Whether messages may be exchanged over this connection
party_id	string	Yes	-	Party identifier used towards the external hub (for MMC-Hub: the receiver EAN on List/Get requests; replaces the former RECEIVER_EAN service setting)
description	string	No	""	Human-readable description

Namespaces are bound to a SOAP connection in two ways: the inbound scheduler uses the connection_id of the schedule entry, and the outbound/acknowledgement paths use the CONNECTION_ID service setting (global default mmchub, overridable per namespace). Because connection documents live under {config_set}/connections/, each config set (DEV/TEST/PROD) carries its own party identifier.

SFTP Connection Structure

{
  "id": string,
  "host": string,
  "port": number,
  "auth": {
    "type": "basic" | "certificate",
    "username": string,           // For type="basic"
    "pw_secret_key": string,      // For type="basic"
    "transport_cert": object,     // For type="certificate"
    "signing_cert": object        // For type="certificate"
  },
  "source_folder": string,
  "allowed_extensions": [string]
}

SFTP Field Reference

Field	Type	Required	Default	Description
id	string	Yes	-	Unique connection identifier
host	string	Yes	-	SFTP server hostname or IP
enabled	boolean	No	true	Whether the connection is polled by the scheduler
port	number	No	22	SFTP port
auth	object	Yes	-	Authentication configuration
auth.type	string	Yes	-	"basic" or "certificate"
auth.username	string	Conditional	-	Username (required if type="basic")
auth.pw_secret_key	string	Conditional	-	Full GCP Secret Manager path for password, e.g. projects/{project_id}/secrets/{secret_name} (required if type="basic")
auth.transport_cert	object	Conditional	-	Transport certificate config (required if type="certificate")
auth.signing_cert	object	Conditional	-	Signing certificate config (required if type="certificate")
source_folder	string	No	null	Folder to retrieve files from
allowed_extensions	array	No	[]	List of allowed file extensions

Certificate Configuration Structure

For type="certificate", each certificate object has:

Field	Type	Required	Default	Description
name	string	Yes	-	Certificate name
data	string	Yes	-	Certificate data (PEM format or secret reference)
expiration_date	string	No	null	Certificate expiration date

SFTP Examples

Basic Authentication:

{
  "id": "acme_sftp",
  "host": "sftp.acme.example.com",
  "port": 22,
  "auth": {
    "type": "basic",
    "username": "energyworx_user",
    "pw_secret_key": "projects/{gcp_project_id}/secrets/acme-sftp-password"
  },
  "source_folder": "/outbound/dre",
  "allowed_extensions": ["xml", "csv"]
}

Certificate Authentication:

{
  "id": "secure_sftp",
  "host": "secure.example.com",
  "port": 22,
  "auth": {
    "type": "certificate",
    "transport_cert": {
      "name": "transport-cert",
      "data": "-----BEGIN CERTIFICATE-----\n...",
      "expiration_date": "2027-12-31"
    },
    "signing_cert": {
      "name": "signing-cert",
      "data": "-----BEGIN CERTIFICATE-----\n...",
      "expiration_date": "2027-12-31"
    }
  },
  "source_folder": "/data",
  "allowed_extensions": ["xml"]
}

Document-pull Connection Structure

A http-pull connection describes a single document published at an HTTP(S) URL. The inbound scheduler emits a trigger per due schedule; the inbound processor downloads the document and forwards it to file manager under the connection's message_type. With change detection enabled (the default) the document is only ingested when its content actually changed: the processor sends conditional requests (If-None-Match / If-Modified-Since) and additionally skips content whose SHA-256 fingerprint was already forwarded.

{
  "id": string,
  "url": string,
  "message_type": string,
  "enabled": boolean,
  "auth": {
    "type": "header" | "param" | "basic",
    "secret_path": string,
    "auth_id": string
  },
  "allowed_content_types": [string],
  "max_file_size_bytes": number,
  "change_detection": boolean,
  "verify_tls": boolean,
  "description": string
}

Document-pull Field Reference

Field	Type	Required	Default	Description
id	string	Yes	-	Unique connection identifier
url	string	Yes	-	Full URL of the document to download
message_type	string	Yes	-	Message type the document is ingested under; must be registered in message_config.json with http_pull_inbound_type: true
enabled	boolean	No	true	Whether the connection is polled by the scheduler
auth	object	No	null	Authentication configuration; omit for public URLs
auth.type	string	Conditional	-	"header" (secret sent as a request header), "param" (as a query parameter) or "basic" (HTTP basic auth)
auth.secret_path	string	Conditional	-	Full GCP Secret Manager path of the credential, e.g. projects/{project_id}/secrets/{secret_name} — the secret value never appears in configuration
auth.auth_id	string	Conditional	-	Header name, query-parameter name, or basic-auth username the secret is sent under
allowed_content_types	array	No	[]	Accepted response Content-Types (empty = accept anything); a response outside the list is rejected without ingestion
max_file_size_bytes	number	No	service default (256 MiB)	Hard cap on the download size; the download is aborted once exceeded
change_detection	boolean	No	true	Skip re-ingesting unchanged content (conditional GET + content fingerprint)
verify_tls	boolean	No	true	Verify the server's TLS certificate
description	string	No	""	Human-readable description

Document-pull Example

{
  "id": "cre_publication_cr",
  "url": "https://www.cre.fr/fileadmin/Documents/Open_data/Transition_energetique_et_appels_d_offres/Publication_donnees_CR.xlsx",
  "message_type": "CRE_PUBLICATION_CR",
  "enabled": true,
  "allowed_content_types": [
    "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
    "application/octet-stream"
  ],
  "change_detection": true,
  "description": "CRE Publication données CR - monthly market data"
}

The message type needs the standard document set of an SFTP-style (body-independent) type: a message_config.json entry with http_pull_inbound_type: true and a rule_configs entry, a field_map_configs/{message_type}.json using only body-independent functions (return_string, generate_uuid, current_date_time, concat_hyphen — set fm_extension to the document's file extension), a response_configs/{message_type}.json, and a rules file (typically {"rules": []}).

---

field_map_configs/{message_type}.json

Defines field extraction and generation for SFTP messages.

Structure

{
  "field_maps": {
    "{field_name}": {
      "function": string,
      "args": any
    }
  }
}

Available Functions

Function	Arguments	Description	Example
return_string	string	Returns static string	{"function": "return_string", "args": "value"}
current_date_time	-	Returns current timestamp	{"function": "current_date_time", "args": ""}
current_date	-	Returns current date	{"function": "current_date", "args": ""}
concat_hyphen	array of functions	Concatenates with hyphen	See example below
get_from_message	xpath_string	Extracts value from XML	{"function": "get_from_message", "args": "//MeterID"}
parse_to_date	xpath_string	Parses a timestamp to a simple date	{"function": "parse_to_date", "args": "CreationTimestamp"}
generate_uuid	-	Generates UUID	{"function": "generate_uuid", "args": ""}
not_null_value	array of functions	Fallback: use the second function if the first returns nothing	See Message Mapper

Standard Fields

Field Name	Required	Description
inbound_filename	Yes	Filename for filemanager storage
message_id	Yes	Unique message identifier
sender_id	Yes	Sender identification
receiver_id	No	Receiver identification
creation_timestamp	Yes	Message creation timestamp

Example

{
  "field_maps": {
    "inbound_filename": {
      "function": "concat_hyphen",
      "args": [
        {"function": "return_string", "args": "dre"},
        {"function": "current_date_time", "args": ""}
      ]
    },
    "message_id": {
      "function": "generate_uuid",
      "args": ""
    },
    "sender_id": {
      "function": "return_string",
      "args": "Acme"
    },
    "receiver_id": {
      "function": "return_string",
      "args": "Energyworx"
    },
    "creation_timestamp": {
      "function": "current_date_time",
      "args": ""
    },
    "meter_id": {
      "function": "get_from_message",
      "args": "//MeterReading/MeterID/text()"
    }
  }
}

---

file_tags/{message_type}.json

Defines tags to be applied to files stored in filemanager.

This file is optional. File tags can also be defined inline in the message map (inbound_file_tags in maps/{message_type}.json), which takes precedence. When neither is configured, files are stored with only the default MessageType and MessageStatus tags.

Structure

{
  "{tag_name}": "{field_reference}"
}

Field Reference

• tag_name: The tag key to apply in filemanager
• field_reference: Reference to field name in field_maps

Example

{
  "SenderID": "sender_id",
  "MessageType": "message_type",
  "ProcessingDate": "creation_timestamp",
  "MeterID": "meter_id",
  "Namespace": "ewx_namespace_id"
}

---

maps/{message_type}.json (SOAP)

Defines field extraction and mapping for SOAP messages. See Message Mapper for detailed documentation.

Special SOAP mapping files

Two files in the maps/ folder have a reserved role and are not per-message-type configurations:

File	Purpose
maps/MetaData.json	Controls the get action (soap_hub.message.soap_actions.get_action). This is a template applied to all message types — {msg_type} is substituted at runtime. Message types that deviate from the common pattern can be overridden by using an object keyed by message type with a default fallback instead of a plain string. This is the only file that affects the GET SOAP action.
maps/ListMap.json	Controls the list action (soap_hub.message.soap_actions.list_action). Applied globally to all list requests; supports the same per-message-type override object as the get action. This is the only file that affects the LIST SOAP action.

The get_action and list_action fields in other per-message-type files (e.g. maps/MeasurementSeriesNotification.json) are ignored for get and list operations. Only send_action and the acknowledgements section are read from per-message-type files.

See Message Mapper — SOAP Actions: Which File to Change for a full breakdown.

---

rules/{message_type}_{process_type}.json

Defines custom validation rules. See Validation Rules for detailed documentation.

---

Configuration Best Practices

1. Naming Conventions

Connection IDs:

• Use lowercase with underscores: acme_sftp, soap_hub_primary
• Include organization and protocol: partner_sftp, markethub_soap

Message Types:

• Use uppercase for acronyms: DRE, GL_MarketDocument
• Be consistent across SOAP and SFTP

Schedule Names:

• Descriptive and unique: acme_dre_schedule, soap_hub_schedule

2. File Organization

• Keep related configurations together
• Use consistent folder structures
• Document custom configurations

3. Version Control

• Store configuration files in a Git repository
• Use separate branches for environments
• Document changes in commit messages
• Review changes before applying to production

4. Secret Management

• Never commit secrets to Git
• Use Secret Manager for all credentials
• Rotate secrets regularly
• Use least-privilege access

5. Testing

• Validate JSON syntax before uploading
• Test in non-production environment first
• Verify configuration with dry-run if possible
• Monitor first deployment closely

---

Configuration Validation

JSON Validation

All configuration files must be valid JSON. Use a validator:

# Validate JSON syntax
jq empty message_config.json

# Pretty-print and validate
jq . schedules/schedule_config.json

Required Fields Check

Before deploying, ensure:

• All required fields are present
• Referenced files exist (schemas, maps, rules)
• Connection IDs match between schedule and connection configs
• Message types exist in message_config.json
• Namespaces are valid and exist
• Secret paths are correct and accessible

---

Troubleshooting Configuration Issues

Configuration Not Loading

Check:

1. File uploaded to correct bucket
2. File naming matches expected pattern
3. JSON syntax is valid
4. Service account has read permissions

Changes Not Taking Effect

Solutions:

1. Restart pods to reload configuration
2. Check configuration cache TTL
3. Verify file timestamp in bucket
4. Review logs for configuration load errors

Schema Validation Failing

Check:

1. XSD file uploaded to schemas folder
2. File path matches message_config.json
3. Schema is valid XSD
4. Namespaces in schema match message

---

Related Documentation

• Implementation Overview
• Message Mapper
• Validation Rules
• SFTP Routing and Scheduling
• Quick Start - SFTP Integration