Data Ingestion

Data ingestion is how measurement data, master data, and market messages enter the Energyworx platform.

Every ingestion path ends at the same place: the data lands in the platform's file manager, a Market Adapter parses it, and a Transformation Configuration maps the parsed output onto datasources, channels, and tags. What differs per path is how the data reaches the platform — who initiates the transfer, over which protocol, and at what cadence.

The platform supports five ingestion methods:

Method	Who initiates	Best for	Protocol
File upload	You (manual) or an external system (automated)	Uploading files: meter reads, master data, market files	File manager UI / automated delivery
Inbound REST API	You (push)	Real-time submission of market messages and measurement series	HTTPS / JSON
Outbound REST polling (RIBS)	Energyworx (pull)	Fetching data from an external API on a schedule or from a flow	HTTPS
Headend system (MODBUS)	Energyworx (pull)	Reading load profiles directly from smart meters	MODBUS
SOAP market hub messaging	Either (hub exchange)	Regulated market communication via SOAP message hubs	SOAP / XML

Choosing a method:

• You have files to ingest (CSV, Excel, XML, JSON) → File upload. Upload them through the file manager in the console, or have an external system deliver them automatically. This is the most common and simplest integration.
• Your system needs to push individual messages in real time and receive validation feedback in the response → Inbound REST API.
• The data lives in a third-party API and Energyworx should go and get it → Outbound REST polling.
• The data has to be collected straight from metering hardware → Headend system.
• You exchange messages with an energy market hub that mandates SOAP → SOAP market hub messaging.

These methods are not mutually exclusive — a typical deployment combines file upload for bulk master data with one of the message-based methods for day-to-day market traffic.

1. File upload

The most common and simplest way to get data in. Files enter through the platform's file manager, where each file is tagged, parsed by a Market Adapter, and ingested. Files can be uploaded by a person through the console, or delivered automatically by an external system.

Through the file manager (console)

This is the normal way for a user to upload files. In the console, go to Smart Integration → File Management and open the upload dialog. There you:

1. Drag and drop one or more files (CSV, Excel, XML, JSON, …).
2. Optionally add tags to attach to the files in the file manager.
3. Optionally select the Market Adapter to parse them with.
4. Toggle Ingest after upload to start ingestion immediately once the files are stored.

This is also the easiest way to test a new Market Adapter or Transformation Configuration before automating delivery. The videos below show the flow end to end.

Automated delivery from an external system

When files need to arrive without anyone using the console, Energyworx can provision a write-only service account that an external system uses to deliver files directly into the storage behind the file manager. Files are placed at a path that encodes the ingestion parameters:

gs://<bucket_name>/path_kwargs/payloadtype=<string>;tags=<a,b,c>/<filename>

Because there is no UI to pick an adapter in this flow, the payloadtype is mapped to the correct Market Adapter through the market_adapters namespace property — see Automated MA Assignment. The tags are attached to the file in the file manager, exactly as with a manual upload.

You configure: a Market Adapter and Transformation Configuration for each payload type, plus the market_adapters namespace property mapping payload types to Market Adapter IDs. Energyworx provides: the bucket name and a write-only service account.

For the full client integration guide — bucket naming, service account setup, and upload examples — see How to upload files to GCS.

2. Inbound REST API

The Integration Service exposes a secure REST endpoint (/v1/erex/aggregated_measurement_series) to which your systems push market messages. Each message is validated on arrival (schema and configurable validation rules) and the validation result is returned synchronously in the response. Messages that pass validation are forwarded to the file manager for ingestion.

You configure: authentication credentials and, where applicable, validation rules for your message types. Use this when: you need synchronous feedback per message, or your data arrives as a continuous stream of individual messages rather than batch files.

See the Inbound Integration RESTful Service documentation for the request format, authentication, and error handling.

3. Outbound REST polling (RIBS)

The REST Invoker Basic System (RIBS) lets the platform fetch data from external APIs. A flow rule calls Ribs.rest_to_gcs(...) with a configured target system and resource path; RIBS handles authentication, performs the request, and lands the response in Cloud Storage for ingestion. Combined with a Trigger Schedule, this enables fully automated periodic collection — for example, fetching daily settlement prices.

You configure: a flow with a rule that calls RIBS, and a trigger schedule for the cadence. Use this when: the data you need is available behind a third-party REST API and the provider will not push it to you.

See the Outbound RESTful Invoker Service documentation for supported target systems, parameters, and code examples.

4. Headend system (MODBUS)

For deployments where the platform reads meters directly, the headend system schedules and executes MODBUS communication with smart meters, collects load profiles, and delivers the results into the ingestion pipeline as JSON.

You configure: the meter inventory and collection schedule. Use this when: there is no intermediate head-end or market party delivering the data — the platform itself is the collection system.

See Simple Headend System MODBUS for the setup guide.

5. SOAP market hub messaging

For markets that communicate through SOAP hubs, the Integration Service implements both patterns:

• Asynchronous: the platform polls the hub for new messages, validates them (sequence, XSD schema, and custom rules), ingests them, and returns acknowledgements. Outbound messages produced by the platform are signed, delivered, and tracked until the hub acknowledges them; failures automatically create tasks in Task Management.
• Synchronous: the platform sends a message and processes the response received in the same session.

You configure: message mappings, XSD schemas, and validation rules in the integration configuration bucket; flows use the XML Message Builder and Publish Outbound Message rules to send messages out. Use this when: the market party you integrate with mandates a SOAP hub.

See the SOAP Services Implementation Overview for the architecture, and How to connect to the Integrations Config Bucket using WinSCP for managing the configuration files.

After ingestion

Whichever path the data takes, the subsequent steps are the same:

1. The file appears in the file manager, tagged with its ingestion parameters.
2. The assigned Market Adapter parses the file and the Transformation Configuration maps it to datasources, channels, and tags.
3. Configured flows run on the ingested data — validation, estimation, and any custom business logic.
4. Progress and failures are visible in Process Monitoring, where each ingested file can be tracked through transformation, storage, and flow execution.

Video walkthroughs

Upload and ingest a master data file:

Upload and ingest a meter data file:

Ingest a meter data file with gaps, and run a flow on the data: