Configuration Reference

This document provides a comprehensive reference for all configuration files used in the Integration Platform.

Configuration Structure Overview

All configuration files are stored in Google Cloud Storage bucket: {project}-integration-service

gs://{project}-integration-service/
├── message_config.json              # Message type definitions
├── schedules/schedule_config.json   # Scheduling and routing configuration
├── connections/                     # Connection configurations
│   ├── {connection_id}.json
│   └── ...
├── field_map_configs/              # SFTP field mapping configurations
│   ├── {message_type}.json
│   └── ...
├── file_tags/                      # File tag configurations
│   ├── {message_type}.json
│   └── ...
├── maps/                           # SOAP field mapping configurations
│   ├── {message_type}.json
│   └── ...
├── rules/                          # Validation rule configurations
│   ├── {message_type}_{process_type}.json
│   └── ...
└── schemas/                        # XSD schemas
    ├── {message_type}.xsd
    └── ...

---

message_config.json

Defines all supported message types for SOAP and SFTP integrations.

Structure

{
  "{MessageType}": {
    "mmchub_inbound_type": boolean,
    "sftp_inbound_type": boolean,
    "xsd_schemas": {
      "code_list": [string],
      "folder": string,
      "file": string
    },
    "message_maps": {
      "folder": string,
      "file": string
    },
    "rule_configs": [
      {
        "process_type_id": string,
        "folder": string,
        "file": string
      }
    ]
  }
}

Field Reference

Field	Type	Required	Default	Description
mmchub_inbound_type	boolean	Yes	-	Enable for SOAP MMC-Hub inbound messages
sftp_inbound_type	boolean	Yes	-	Enable for SFTP inbound messages
xsd_schemas	object	No		XSD schema configuration for validation
xsd_schemas.code_list	array	No	[]	List of XSD codelist files
xsd_schemas.folder	string	No	-	Folder containing schema files
xsd_schemas.file	string	No	-	Main XSD schema filename
message_maps	object	No		SOAP field mapping configuration
message_maps.folder	string	No	-	Folder containing mapping files
message_maps.file	string	No	-	Mapping configuration filename
rule_configs	array	No	[]	Validation rule configurations
rule_configs[].process_type_id	string	Yes	-	Unique process type identifier
rule_configs[].folder	string	Yes	-	Folder containing rule files
rule_configs[].file	string	Yes	-	Rule configuration filename

Examples

SOAP Message with Validation:

{
  "GL_MarketDocument": {
    "mmchub_inbound_type": true,
    "sftp_inbound_type": false,
    "xsd_schemas": {
      "code_list": ["urn-entsoe-eu-wgedi-codelists.xsd"],
      "folder": "schemas",
      "file": "GL_MarketDocument.xsd"
    },
    "message_maps": {
      "folder": "maps",
      "file": "MarketDocument.json"
    },
    "rule_configs": [
      {
        "process_type_id": "A26",
        "folder": "rules",
        "file": "GL-MarketDocument_A26.json"
      }
    ]
  }
}

Simple SFTP Message:

{
  "DRE": {
    "sftp_inbound_type": true,
    "xsd_schemas": {},
    "message_maps": {},
    "rule_configs": []
  }
}

---

schedules/schedule_config.json

Defines scheduling configuration for SOAP and SFTP message retrieval.

Structure

{
  "schedule_types": {
    "{schedule_name}": {
      "type": string,
      "description": string,
      "interval": string,
      "timezone": string
    }
  },
  "schedules": [
    {
      "name": string,
      "enabled": boolean,
      "type": string,
      "connection_id": string,
      "schedule": string,
      "namespaces": [string],
      "description": string
    }
  ]
}

Schedule Types Field Reference

Field	Type	Required	Default	Description
type	string	Yes	-	"interval", "manual", or "disabled"
description	string	No	""	Human-readable description
interval	string	Conditional	-	Interval as minutes or hours, e.g. "1m", "15m", "1h" (if type="interval")
timezone	string	No	"UTC"	Timezone for schedule execution

Schedules Field Reference

Field	Type	Required	Default	Description
name	string	Yes	-	Unique schedule identifier
enabled	boolean	Yes	-	Enable/disable the schedule
type	string	Yes	-	"soap" or "sftp"
connection_id	string	Yes	-	Reference to connection config
schedule	string	Yes	-	Reference to schedule_type name
namespaces	array	Yes	-	List of namespace IDs to process
description	string	No	""	Human-readable description

Examples

{
  "schedule_types": {
    "every_minute": {
      "type": "interval",
      "description": "Run every minute",
      "interval": "1m",
      "timezone": "UTC"
    },
    "every_5_minutes": {
      "type": "interval",
      "description": "Run every 5 minutes",
      "interval": "5m",
      "timezone": "UTC"
    }
  },
  "schedules": [
    {
      "name": "soap_hub_schedule",
      "enabled": true,
      "type": "soap",
      "connection_id": "soap_hub_primary",
      "schedule": "every_minute",
      "namespaces": ["enrx_org_004", "enrx_org_006"],
      "description": "Check the SOAP hub for new messages"
    },
    {
      "name": "acme_sftp_schedule",
      "enabled": true,
      "type": "sftp",
      "connection_id": "acme_sftp",
      "schedule": "every_5_minutes",
      "namespaces": ["enrx_org_004"],
      "description": "Process DRE files from Acme SFTP"
    }
  ]
}

---

connections/{connection_id}.json

Defines SFTP or SOAP connection configurations.

SFTP Connection Structure

{
  "id": string,
  "host": string,
  "port": number,
  "auth": {
    "type": "basic" | "certificate",
    "username": string,           // For type="basic"
    "pw_secret_key": string,      // For type="basic"
    "transport_cert": object,     // For type="certificate"
    "signing_cert": object        // For type="certificate"
  },
  "source_folder": string,
  "allowed_extensions": [string]
}

SFTP Field Reference

Field	Type	Required	Default	Description
id	string	Yes	-	Unique connection identifier
host	string	Yes	-	SFTP server hostname or IP
port	number	No	22	SFTP port
auth	object	Yes	-	Authentication configuration
auth.type	string	Yes	-	"basic" or "certificate"
auth.username	string	Conditional	-	Username (required if type="basic")
auth.pw_secret_key	string	Conditional	-	Full GCP Secret Manager path for password, e.g. projects/{project_id}/secrets/{secret_name} (required if type="basic")
auth.transport_cert	object	Conditional	-	Transport certificate config (required if type="certificate")
auth.signing_cert	object	Conditional	-	Signing certificate config (required if type="certificate")
source_folder	string	No	null	Folder to retrieve files from
allowed_extensions	array	No	[]	List of allowed file extensions

Certificate Configuration Structure

For type="certificate", each certificate object has:

Field	Type	Required	Default	Description
name	string	Yes	-	Certificate name
data	string	Yes	-	Certificate data (PEM format or secret reference)
expiration_date	string	No	null	Certificate expiration date

SFTP Examples

Basic Authentication:

{
  "id": "acme_sftp",
  "host": "sftp.acme.example.com",
  "port": 22,
  "auth": {
    "type": "basic",
    "username": "energyworx_user",
    "pw_secret_key": "projects/{gcp_project_id}/secrets/acme-sftp-password"
  },
  "source_folder": "/outbound/dre",
  "allowed_extensions": ["xml", "csv"]
}

Certificate Authentication:

{
  "id": "secure_sftp",
  "host": "secure.example.com",
  "port": 22,
  "auth": {
    "type": "certificate",
    "transport_cert": {
      "name": "transport-cert",
      "data": "-----BEGIN CERTIFICATE-----\n...",
      "expiration_date": "2027-12-31"
    },
    "signing_cert": {
      "name": "signing-cert",
      "data": "-----BEGIN CERTIFICATE-----\n...",
      "expiration_date": "2027-12-31"
    }
  },
  "source_folder": "/data",
  "allowed_extensions": ["xml"]
}

---

field_map_configs/{message_type}.json

Defines field extraction and generation for SFTP messages.

Structure

{
  "field_maps": {
    "{field_name}": {
      "function": string,
      "args": any
    }
  }
}

Available Functions

Function	Arguments	Description	Example
return_string	string	Returns static string	{"function": "return_string", "args": "value"}
current_date_time	-	Returns current timestamp	{"function": "current_date_time", "args": ""}
current_date	-	Returns current date	{"function": "current_date", "args": ""}
concat_hyphen	array of functions	Concatenates with hyphen	See example below
get_from_message	xpath_string	Extracts value from XML	{"function": "get_from_message", "args": "//MeterID"}
parse_to_date	xpath_string	Parses a timestamp to a simple date	{"function": "parse_to_date", "args": "CreationTimestamp"}
generate_uuid	-	Generates UUID	{"function": "generate_uuid", "args": ""}
not_null_value	array of functions	Fallback: use the second function if the first returns nothing	See Message Mapper

Standard Fields

Field Name	Required	Description
inbound_filename	Yes	Filename for filemanager storage
message_id	Yes	Unique message identifier
sender_id	Yes	Sender identification
receiver_id	No	Receiver identification
creation_timestamp	Yes	Message creation timestamp

Example

{
  "field_maps": {
    "inbound_filename": {
      "function": "concat_hyphen",
      "args": [
        {"function": "return_string", "args": "dre"},
        {"function": "current_date_time", "args": ""}
      ]
    },
    "message_id": {
      "function": "generate_uuid",
      "args": ""
    },
    "sender_id": {
      "function": "return_string",
      "args": "Acme"
    },
    "receiver_id": {
      "function": "return_string",
      "args": "Energyworx"
    },
    "creation_timestamp": {
      "function": "current_date_time",
      "args": ""
    },
    "meter_id": {
      "function": "get_from_message",
      "args": "//MeterReading/MeterID/text()"
    }
  }
}

---

file_tags/{message_type}.json

Defines tags to be applied to files stored in filemanager.

This file is optional. File tags can also be defined inline in the message map (inbound_file_tags in maps/{message_type}.json), which takes precedence. When neither is configured, files are stored with only the default MessageType and MessageStatus tags.

Structure

{
  "{tag_name}": "{field_reference}"
}

Field Reference

• tag_name: The tag key to apply in filemanager
• field_reference: Reference to field name in field_maps

Example

{
  "SenderID": "sender_id",
  "MessageType": "message_type",
  "ProcessingDate": "creation_timestamp",
  "MeterID": "meter_id",
  "Namespace": "ewx_namespace_id"
}

---

maps/{message_type}.json (SOAP)

Defines field extraction and mapping for SOAP messages. See Message Mapper for detailed documentation.

Special SOAP mapping files

Two files in the maps/ folder have a reserved role and are not per-message-type configurations:

File	Purpose
maps/MetaData.json	Controls the get action (soap_hub.message.soap_actions.get_action). This is a template applied to all message types — {msg_type} is substituted at runtime. Message types that deviate from the common pattern can be overridden by using an object keyed by message type with a default fallback instead of a plain string. This is the only file that affects the GET SOAP action.
maps/ListMap.json	Controls the list action (soap_hub.message.soap_actions.list_action). Applied globally to all list requests; supports the same per-message-type override object as the get action. This is the only file that affects the LIST SOAP action.

The get_action and list_action fields in other per-message-type files (e.g. maps/MeasurementSeriesNotification.json) are ignored for get and list operations. Only send_action and the acknowledgements section are read from per-message-type files.

See Message Mapper — SOAP Actions: Which File to Change for a full breakdown.

---

rules/{message_type}_{process_type}.json

Defines custom validation rules. See Validation Rules for detailed documentation.

---

Configuration Best Practices

1. Naming Conventions

Connection IDs:

• Use lowercase with underscores: acme_sftp, soap_hub_primary
• Include organization and protocol: partner_sftp, markethub_soap

Message Types:

• Use uppercase for acronyms: DRE, GL_MarketDocument
• Be consistent across SOAP and SFTP

Schedule Names:

• Descriptive and unique: acme_dre_schedule, soap_hub_schedule

2. File Organization

• Keep related configurations together
• Use consistent folder structures
• Document custom configurations

3. Version Control

• Store configuration files in a Git repository
• Use separate branches for environments
• Document changes in commit messages
• Review changes before applying to production

4. Secret Management

• Never commit secrets to Git
• Use Secret Manager for all credentials
• Rotate secrets regularly
• Use least-privilege access

5. Testing

• Validate JSON syntax before uploading
• Test in non-production environment first
• Verify configuration with dry-run if possible
• Monitor first deployment closely

---

Configuration Validation

JSON Validation

All configuration files must be valid JSON. Use a validator:

# Validate JSON syntax
jq empty message_config.json

# Pretty-print and validate
jq . schedules/schedule_config.json

Required Fields Check

Before deploying, ensure:

• All required fields are present
• Referenced files exist (schemas, maps, rules)
• Connection IDs match between schedule and connection configs
• Message types exist in message_config.json
• Namespaces are valid and exist
• Secret paths are correct and accessible

---

Troubleshooting Configuration Issues

Configuration Not Loading

Check:

1. File uploaded to correct bucket
2. File naming matches expected pattern
3. JSON syntax is valid
4. Service account has read permissions

Changes Not Taking Effect

Solutions:

1. Restart pods to reload configuration
2. Check configuration cache TTL
3. Verify file timestamp in bucket
4. Review logs for configuration load errors

Schema Validation Failing

Check:

1. XSD file uploaded to schemas folder
2. File path matches message_config.json
3. Schema is valid XSD
4. Namespaces in schema match message

---

Related Documentation

• Implementation Overview
• Message Mapper
• Validation Rules
• SFTP Routing and Scheduling
• Quick Start - SFTP Integration