19 docs tagged with "Ingestion"

Complete field-by-field reference for all integration service configuration files: message config, schedules, connections, field maps, and file tags.

Ingest measurement data into the Energyworx platform via file upload, REST APIs, polling, MODBUS, or SOAP market hub messaging.

Connect to the GCP integrations configuration bucket using WinSCP with service account credentials for uploading configs.

Download files from Energyworx storage using the API v1.5 with Bearer access tokens and custom search parameters.

Provision identity-service client credentials for the integration services, store them SOPS-encrypted, and understand the runtime OAuth2 token flow.

Upload message files directly to the Energyworx GCS bucket with service account credentials and path-based ingestion parameters.

Implement asynchronous and synchronous SOAP message processing and SFTP file-based integrations for energy market communications.

Process and validate incoming measurement data through a secure REST API with automated validation and audit trails.

Configure and implement Market Adapters to parse incoming files and normalize data for ingestion into the Energyworx platform.

Configure SOAP and SFTP message types, XSD schemas, mappings, and rule configs in the integration service config bucket.

Map SOAP and SFTP messages to a common message object for routing, validation, and acknowledgement processing.

Route inbound SOAP messages to the correct namespace using receiver EAN credentials and scheduled polling.

Fetch external market data like reconciliation prices and store responses to Google Cloud Storage automatically.

Step-by-step guide to add a new SFTP integration: secrets, connection config, schedules, message types, and go-live checks.

Define header and field validation rules in JSON for SOAP messages with error codes and custom validation logic.

Route and schedule SFTP-based inbound messages using schedule and connection configurations mapping connections to namespaces.

Automate meter data collection from SEL smart meters using MODBUS protocol with the Headend System scheduler and processor.

Map parsed data to datasources, channels, and tags using drag-and-drop transformation configuration with optional flow design integration.

Schedule automated processes using cron expressions to run recurring flows, aggregations, and BigQuery ingestion jobs.