Uploading files to Energyworx — Client integration guide

This document describes how an external Client uploads message files to Energyworx for automated ingestion. Files are delivered directly to a Google Cloud Storage (GCS) bucket owned by Energyworx, using a service account that Energyworx provisions for the Client.

The bucket acts as the entry point for the Energyworx platform's file manager — any file dropped under the right prefix is picked up and routed to the correct market adapter automatically.

Throughout this document, "the Client" refers to the integrating party. Replace it with the actual client or partner name when this document is customised per integration.

1. General setup

In order for Energyworx to process data from the Client, the Client uploads message files directly into an Energyworx-managed GCS bucket. The bucket is the same bucket the platform's file manager uses, so uploads do not require any additional API calls — placing the file at the right object path is enough to trigger ingestion.

Each file must be uploaded to a path that not only specifies the location but also carries the parameters that tell the platform how the file should be processed.

The expected object path looks like this:

gs://<bucket_name>/path_kwargs/payloadtype=<string>;tags=<a,b,c>/<filename>

Path components

gs://<bucket_name> — the GCS bucket files must be uploaded to. The bucket name follows the pattern <shortenv>-namespace-<namespace_id> (for example, clientstg-namespace-org_001 for the Client's development namespace/environment). Energyworx provides the exact name for each environment (typically one for staging/acceptance and one for production).
/path_kwargs — a reserved top-level folder in the bucket. Files placed under this folder are picked up by the Energyworx platform and ingested automatically according to the parameters encoded in the next path segment.
payloadtype=<string>;tags=<a,b,c> — the ingestion parameters, encoded as a single path segment. The accepted fields are:
- payloadtype — a hard-coded string identifying the message type. The platform uses a namespace property called market_adapters to map the payloadtype value to the correct market adapter, so the market adapter does not need to be specified directly. Required.
- tags — a comma-separated list of tags that should be attached to the file in the file manager. Optional.
/<filename> — the actual file name. Use a unique, descriptive name (for example, one that includes a message ID and timestamp) so that uploads are idempotent and traceable.

Example

gs://clientstg-namespace-org_001/path_kwargs/payloadtype=<payloadtype>;tags=<tag1>,<tag2>/2026-05-20T08-15-00Z_msg-7a3f.xml

2. Prerequisites

Before the Client can upload, the following information needs to be requested through the ServiceDesk:

The bucket name for each environment (acceptance and production).
A service account email of the form <client>-uploader@<ewx-project>.iam.gserviceaccount.com.
A service account key file (JSON) for that service account.
The payload type string(s) the Client should use in the path. The platform maps these to the correct market adapter internally.

The service account is granted only the minimum permissions needed to write objects to the agreed bucket — specifically roles/storage.objectCreator on the bucket. It cannot list, read or delete objects, and it has no permissions on any other resource. It can, however, overwrite an existing object name — see the note in section 6.

3. Authentication with a service account

All uploads must be authenticated using the service account key provided by Energyworx. The key is a JSON file that looks roughly like this (values shortened for readability):

{
  "type": "service_account",
  "project_id": "ewx-...",
  "private_key_id": "...",
  "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
  "client_email": "<client>-uploader@ewx-....iam.gserviceaccount.com",
  "client_id": "...",
  "token_uri": "https://oauth2.googleapis.com/token"
}

Handling the key safely

Treat the key file as a secret. Anyone with the file can upload to the Energyworx bucket on the Client's behalf.
Store it in a secret manager (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Kubernetes Secret, etc.) — never commit it to source control.
Restrict file permissions on disk (chmod 600) if it must be stored on a server.
Rotate the key periodically. Energyworx can issue a new key on request; notify Energyworx before retiring the old one so it can be removed from IAM.

Permissions and OAuth scope — what to expect

At the IAM level the service account is write-only: roles/storage.objectCreator grants only storage.objects.create. Read, list and delete calls all return 403 Forbidden by design.

GCP does not publish a write-only Cloud Storage OAuth scope, so the recommended scope when minting a token is devstorage.read_write. The token therefore carries read capability in principle, but the actual access is the intersection of scope and IAM — IAM is the binding constraint, so read attempts still fail. There is no need to be concerned that the wider scope grants unintended access.

storage.objects.create does allow writing to an existing object name — i.e. overwriting the live version of an object. It does not grant delete, so old versions are preserved if the bucket has Object Versioning enabled. The recommended mitigation is to use unique object names (timestamp + message ID, as shown in the examples).

Two ways to authenticate

The recommended approach is to point an SDK or CLI tool at the key file using the GOOGLE_APPLICATION_CREDENTIALS environment variable. The SDK takes care of signing a JWT, exchanging it for an OAuth2 access token, and refreshing the token as needed.

export GOOGLE_APPLICATION_CREDENTIALS=/secure/path/client-uploader.json

If a Google SDK cannot be used (for example, when uploading from a system that only has curl), the access token can be minted manually from the key. See section 5 below.

4. Uploading with Python (`google-cloud-storage`)

This is the recommended approach for any environment that can run Python.

Install

pip install google-cloud-storage

Upload a file

import os
from google.cloud import storage

# Either set GOOGLE_APPLICATION_CREDENTIALS in the environment, or pass
# the key file explicitly:
#   client = storage.Client.from_service_account_json("/secure/path/client-uploader.json")
client = storage.Client("<ewx-project-id>")

BUCKET_NAME = "<shortenv>-namespace-<namespace_id>"  # provided by Energyworx

def upload_message(
    local_path: str,
    filename: str,
    payloadtype: str,
    tags: list[str] | None = None,
) -> str:
    """Upload a message file to the Energyworx ingestion bucket.

    The platform resolves the market adapter from `payloadtype`.
    Returns the gs:// URI the file was uploaded to.
    """
    if not payloadtype:
        raise ValueError("payloadtype is required")

    parts = [f"payloadtype={payloadtype}"]
    if tags:
        parts.append("tags=" + ",".join(tags))

    kwargs_segment = ";".join(parts)
    object_path = f"path_kwargs/{kwargs_segment}/{filename}"

    bucket = client.bucket(BUCKET_NAME)
    blob = bucket.blob(object_path)
    blob.upload_from_filename(local_path)

    return f"gs://{BUCKET_NAME}/{object_path}"


if __name__ == "__main__":
    uri = upload_message(
        local_path="./outbox/msg-7a3f.xml",
        filename="2026-05-20T08-15-00Z_msg-7a3f.xml",
        payloadtype="<payloadtype>",
        tags=["<tag1>", "<tag2>"],
    )
    print("Uploaded to:", uri)

Notes

The client picks up GOOGLE_APPLICATION_CREDENTIALS automatically; the key file does not need to be passed in code. Use storage.Client.from_service_account_json(...) only if the key must be loaded from a non-standard location.
upload_from_filename performs a resumable upload for larger files and a single-shot upload for small ones. No additional configuration is required.
The service account does not have storage.objects.get, so do not call blob.exists() or blob.reload() after uploading — those calls will return a 403. A successful return from upload_from_filename is the confirmation that the file was accepted.

5. Uploading with `curl` (REST API)

If a full SDK is not available, the Client can upload directly via the JSON API. An OAuth2 access token must be minted from the service account key first.

Step 1 — Mint an access token

Google's recommended way to do this without an SDK is to sign a JWT with the service account's private key and exchange it for an access token at https://oauth2.googleapis.com/token

The easiest tool for this in a shell environment is oauth2l github.com/google/oauth2l or gcloud. For example:

# Using gcloud (requires the Google Cloud SDK to be installed)
gcloud auth activate-service-account \
  --key-file=/secure/path/client-uploader.json

ACCESS_TOKEN=$(gcloud auth print-access-token)

# Using oauth2l
ACCESS_TOKEN=$(oauth2l fetch \
  --credentials /secure/path/client-uploader.json \
  --scope https://www.googleapis.com/auth/devstorage.read_write)

If neither tool is available, the JWT can be constructed and signed in any language with an RSA-SHA256 library; consult Google's service account JWT flow documentation for the exact claims.

The access token is valid for one hour. Re-mint it before it expires.

Step 2 — Upload the file

Use the JSON API's media upload endpoint. The object name must NOT be URL-encoded.

BUCKET=<shortenv>-namespace-<namespace_id>
FILENAME=2026-05-20T08-15-00Z_msg-7a3f.xml
PATH_KWARGS="payloadtype=<payloadtype>;tags=<tag1>,<tag2>"

# The object name.
OBJECT_NAME="path_kwargs/${PATH_KWARGS}/${FILENAME}"

curl -X POST \
  -H "Authorization: Bearer ${ACCESS_TOKEN}" \
  -H "Content-Type: application/xml" \
  --data-binary @./outbox/msg-7a3f.xml \
  "https://storage.googleapis.com/upload/storage/v1/b/${BUCKET}/o?uploadType=media&name=${OBJECT_NAME}"

A successful response is HTTP 200 with a JSON body describing the new object. Any non-2xx response indicates the upload was not accepted — see the troubleshooting section.

Notes

Set Content-Type to the actual media type of the message (application/xml, application/json, text/csv, …). It is stored as the object's content type and surfaced in the file manager.
uploadType=media is appropriate for files up to a few MB. For larger files, use uploadType=resumable; see performing-resumable-uploads.
Do not URL-encode the object name. Pass it verbatim as the name query parameter — including the /, ; and = characters — exactly as shown above. GCS interprets the raw value as the object name.

6. Verifying the upload

After a successful upload:

The file appears in the Energyworx file manager within seconds, tagged with the values from the tags= field.
The platform routes the file to the market adapter resolved from the payloadtype value.
If ingestion fails (for example, because of a malformed message), the failure is visible in the Energyworx file manager and can be investigated by Energyworx support.

The Client's service account does not have permission to list or read objects in the bucket, so success cannot be confirmed by re-reading the file. Treat an HTTP 2xx (or a clean return from the Python client) as confirmation that GCS accepted the object.

Overwrite caveat. Because storage.objects.create permits writes to an existing object name, uploading the same filename twice replaces the live object. Use unique filenames (timestamp + message ID is the recommended pattern) so retries and replays are idempotent rather than destructive. If overwrite protection is required, request that Energyworx enables Object Versioning on the bucket.

7. Troubleshooting

Symptom	Likely cause	Resolution
`401 Unauthenticated`	Access token missing, malformed or expired.	Re-mint the access token. Verify `GOOGLE_APPLICATION_CREDENTIALS` points to the correct file.
`403 Forbidden` on upload	The service account lacks `storage.objects.create` on the bucket, the bucket name is wrong, or the wrong service account key is being used.	Confirm the bucket name with Energyworx. Confirm the `client_email` in the key matches the one Energyworx provisioned.
`403 Forbidden` on `GET`/`list`	Expected. The service account is write-only by design.	Do not perform read operations; rely on the `2xx` response from the upload call as confirmation.
`404 Not Found` on upload	Bucket name is misspelled or the wrong environment's bucket is being used.	Double-check the bucket name.
Upload succeeds but the file never appears in the file manager	The object was placed outside `path_kwargs/`, or the kwargs segment is malformed (missing `payloadtype` or wrong separators).	Verify the full object path matches the convention in section 1.
File appears but is tagged or routed incorrectly	`payloadtype` or `tags` is misspelled or has unexpected whitespace, or the `payloadtype` is not registered in the `market_adapters` namespace.	Check the path segment exactly — separator is `;`, key/value separator is `=`, list separator inside `tags=` is `,`. No spaces. Confirm the `payloadtype` value with Energyworx.

8. Support

For bucket names, service account provisioning, key rotation, and the payload type strings registered for the Client, please reach out to the designated Energyworx contact. Include the environment (acceptance / production) and the service account email in any support requests.

1. General setup​

Path components​

Example​

2. Prerequisites​

3. Authentication with a service account​

Handling the key safely​

Permissions and OAuth scope — what to expect​

Two ways to authenticate​

4. Uploading with Python (google-cloud-storage)​

Install​

Upload a file​

Notes​

5. Uploading with curl (REST API)​

Step 1 — Mint an access token​

Step 2 — Upload the file​

Notes​

6. Verifying the upload​

7. Troubleshooting​

8. Support​

1. General setup

Path components

Example

2. Prerequisites

3. Authentication with a service account

Handling the key safely

Permissions and OAuth scope — what to expect

Two ways to authenticate

4. Uploading with Python (`google-cloud-storage`)

Install

Upload a file

Notes

5. Uploading with `curl` (REST API)

Step 1 — Mint an access token

Step 2 — Upload the file

Notes

6. Verifying the upload

7. Troubleshooting

8. Support