What's New ✨​

Dynamic datasource selection for scheduled flows​

Scheduled flow trigger schedules can now select their datasources with a query instead of a fixed list. A scheduled flow can target a self-maintaining set of datasources — for example, every datasource whose name starts with a given prefix — resolved fresh at each run, so datasources added or removed later are picked up automatically. See the Trigger Schedule documentation.

Channel names​

Channel classifiers can now show a different display name per datasource. The display name is set through a per-datasource channel name, which you can configure via a custom Market Adapter, a Transformation Configuration, or directly from a rule.

Analytics dashboards (beta)​

A beta version of Analytics dashboards — a new dashboarding tool with additional capabilities — is now available on request, at additional cost. Your existing native dashboards are unaffected and continue to work as expected. This is our first trial of the tool; if you'd like to try it, contact our service desk.

⚠️ Deprecation Notice​

• X-API-KEY authentication deprecated: Authenticating API requests with the X-API-KEY header is deprecated and will be removed in the next release (26.09), as this authentication method is being retired for security reasons. If needed, migrate to OAuth2 client credentials issued through the new console; requests that still use X-API-KEY now log a deprecation warning on the platform side. Requests using Authorization will continue to work.
• Analyze command deprecation: The 'ewx-cli analyze' command family (rule and flow complexity analysis) is deprecated and will be removed in release 27.02. Invoking any analyze command now prints a deprecation warning.
• API v2 organization/project resource paths: The organization/project prefix in API v2 resource paths is deprecated and will be removed in release 27.02 — use the shorter namespaces/{namespace}/… form instead.

🖥️ UI Improvements​

• Dashboard action renamed: The dashboard-creation action in the new UI is now labeled "Add Dashboard" instead of "Create Dashboard", in all supported languages. (New console)
• Audit event search fix: Audit event searches in the console now cover the selected dates in your own timezone instead of UTC, so events logged near midnight are no longer missing from the day they belong to. The severity counters on the datasource details page follow the same range. (New console)
• Predefined values: You can now add new predefined values to an existing enum data point attribute when editing a channel classifier. Previously these values were read-only once the classifier had been created. (New console)
• Custom separators in Transformation Configurations: Fixed an issue where custom separators for merged columns were ignored when creating a Transformation Configuration via the UI for tag names, channel attributes, and flow properties. (New console)
• Correct default timeseries date range: Fixed an issue where the default timeseries date range was incorrectly calculated from midnight in the Original UI and in UTC in the New UI. The date range now correctly counts from the current time and respects the user's selected timezone. (Both consoles)
• Duplicate channel-family names now show an error: Fixed an issue where creating a channel family with an already existing technical name would fail silently without displaying an error message. (New console)
• Notifications: Error and info messages are no longer cut off behind the project documentation panel — when the panel is open, notifications shift alongside it and stay fully visible. (New console)
• Search state preserved on navigation: The datasource search query and filters are now retained when navigating to a result's Details or Flows and back. (New console)
• New console fixes: Resolved a range of UI issues across the new console, including dates now shown in your local timezone, full-name tooltips on truncated text, clearer action icons, better handling of long tag lists, loading feedback on task boards, and faster, smoother datasource search. (New console)
• Faster analytics dashboard loads: Fixed an issue where opening an analytics dashboard would issue a redundant backend request, improving load times. (New console)
• Easier date & time filters: Made the date & time filters on the Audit Events and File Management pages easier to use: the native time spinner is replaced with a calendar for the date plus hour/minute(/second) fields you can either type into or pick from a dropdown. (New console)
• Permissions table clarity: Renamed the "Computedflag" permission resource to "Live Flags", removed the redundant "System" resource, and added clearer tooltips describing each resource (including how Timeseries and Datapoint differ, and what the Integration and Documentation resources control). (Original console)
• Annotations show even without datapoints: Fixed an issue where annotations were not shown in charts if there were no related datapoints for the channel. (New console)
• Multi-line task descriptions: Task descriptions now keep their line breaks in the taskboard, so multi-line and structured bodies (such as automated data-quality reports) stay readable instead of collapsing onto a single line. (New console)
• Scrolling past charts: Scrolling the mouse wheel over a chart no longer zooms it, so the page scrolls normally when a chart is under the cursor. Use click-and-drag on the chart or the range selector buttons to zoom into a period. (New console)
• Edit datasource classifiers: Existing datasource classifiers can now be edited in the new UI — description, datasource type, and source type. (New console)

⚙️ Custom Business Logic​

• Non-existent Pub/Sub topics now raise a warning: When a flow forwards a message to a Pub/Sub topic that does not exist (for example a typo in the pubsub_topic_name namespace property), the audit events now show a warning naming the topic instead of reporting success. The flow still runs to completion, and the message is no longer retried indefinitely.
• Channel rename from a rule: Renaming a channel from a rule now reliably persists, fixing an intermittent case where the channel could keep its previous name.
• Easier testing for custom rules: The ewx-public package now includes the full toolkit for unit-testing custom rules — synthetic time-series data generators (monotonic, sinusoidal, and seasonal series, register channels, and more), the shared rule test base class, and the ability to build and inspect rules and tags directly in your tests. Teams can move existing tests off the legacy package with only a change to the import line, and invalid tags now fail a test instead of silently passing with a warning.
• Custom-rule helpers load on a fresh install: The date-parsing and holiday-calendar helpers for custom rules no longer fail to load on a fresh install because of a missing dependency; they now work out of the box.
• Direct imports from standard-packages: Custom rules and scripts can now import the shared helper modules from the standard-packages library directly (for example StandardUtils) without running into import errors.
• Change channel names from rules: Rules can now rename a channel on a datasource with set_channel_name(). Previously a channel name could only be set during ingestion, through a Market Adapter or Transformation Configuration.

🔐 API & Permissions​

• API v2 tag search: Datasource search through API v2 now finds datasources by a tag property alone, such as city=Houten, instead of returning no results. Previously only the fully qualified form location.city=Houten matched, while API v1 already supported both.
• Channel classifier consistency: Channel classifier attributes now behave consistently across API v1 and v2. The technical name is immutable, classifier ids may contain the '+' character, and the per-datasource channel name is optional.
• User API response hardening: User detail and user list API responses no longer include the apiKeyToken, refreshToken and keyfile credential fields. Retrieving a keyfile remains available through the dedicated user keyfile endpoint.
• Fixed access for users with mixed-case email addresses: Users whose identity provider (e.g. Microsoft Entra) returns their email with capital letters were incorrectly denied access to dashboards and other platform features. Authorization now matches the stored account regardless of email casing.

📚 Documentation Updates​

• Integration service documentation refresh: The integration service documentation now covers SFTP-based integrations alongside SOAP, including a step-by-step quick start guide, routing and scheduling instructions, and a complete configuration reference. The existing SOAP documentation has also been updated to reflect current configuration options and acknowledgement mapping, and the architecture overview now includes integration services.
• Documentation capability overview: Added a Platform Capabilities overview page and new guides for datasource search, flow orchestration, anomaly detection and alerting, and multi-tenancy and namespaces. Existing pages on market communication, billing determinants and virtual datasources now open with clearer business context.
• Standard packages documentation: The shareable-packages section now documents the standard EDM packages (standard-packages) — what they contain, how to install them from the restricted Artifact Registry, and how to deploy a package to a namespace with ewx-cli. The packages are a separately licensed offering, available only to organizations with a standard-packages agreement.

🚀 Notable Backend Changes​

• Security hardening: This release bundles a broad set of security and dependency updates across the platform, gateway, integration, and documentation-site services — patching known CVEs in cryptography, networking, SAML, TLS/JWT, gRPC authorization, and other libraries, hardening and slimming container images, and strengthening the automated dependency-update (Renovate) and vulnerability-scanning pipelines.
• Process Monitoring instance counts: Fixed several issues that could make a Process Monitoring configuration's instance counts inaccurate under heavy or repeated processing — the total and successful counts could end up slightly too high or too low, and an instance could stay marked "in progress" after it had already finished. Counts are now accurate and the in-progress count settles to zero once processing completes.
• Correct promo-metric reset across timezones: Promo metrics now reset at local midnight for timezones with fractional-hour offsets (e.g. India, Central Australia, Chatham Islands).
• Classifier filter finds new datasources: Datasource search now correctly returns results when filtering by classifier for newly-added datasources.
• Namespace-relative APIv2 resource names: APIv2 data endpoints now accept namespace-relative resource paths (namespaces/{namespace}/...) alongside the existing organizations/{organization}/projects/{project}/namespaces/{namespace}/... form, and responses now return resource names in the shorter namespace-relative form. The organization/project prefix is deprecated: it keeps working during the transition and will be removed in the 27.02 release. The web console and ewx-cli have moved to the namespace-relative paths. OAuth2 client-credential resources remain organization-scoped and are unaffected.
• Datasource search fix: A datasource that had its classifier changed no longer appears twice in search results; it is now returned only under its current classifier.
• Removed tags no longer searchable: Searching by a tag that has been removed from a datasource no longer returns that datasource; only its current tags are indexed for search.
• Lower CPU on large timeseries loads: Fixed an issue that could cause high CPU usage when loading large amounts of timeseries data.
• Scheduled flow triggers can select datasources by query: The flow trigger schedule now accepts a datasource_ids_query (a BigQuery SELECT whose first column is a datasource id) as an alternative to a static datasource_ids list, resolved at execution time so the flow runs on whatever datasources currently match — letting a scheduled flow target a dynamic, self-maintaining set (e.g. every datasource whose name starts with a given prefix) with no id list to maintain.
• Behind-the-scenes platform modernization — no customer impact: A significant part of this release is internal engineering to standardize how platform services store and retrieve their configuration and metadata — spanning datasources, classifiers, rules, flows, triggers, transformation configurations, market adapters, namespaces, billing accounts, ML models, and tasks — on a single shared foundation. This is preparatory groundwork only: there is no change to how the platform behaves, no change to your data or APIs, and no action required on your part.

📡 Integration Component​

• SFTP reconnection fix: The integration component now transparently reconnects when an SFTP connection becomes stale due to a server-side idle timeout, preventing file processing failures.
• N243 validation fix: N243 settlement result messages for the same grid delivery station and allocation period are now validated per market participant (BRP), so responses for different BRPs are no longer incorrectly rejected as duplicates with error 704.
• Unified integration messaging and logging: The SOAP, REST, Modbus, and RIBS integration services now run on the same messaging and logging foundations as the rest of the platform. Message delivery is unchanged — including handling of very large payloads — but every message now carries a standard identifier for end-to-end tracing, and all four services emit consistent, structured logs. The result is integration activity that's easier to monitor and faster to troubleshoot, with no configuration changes required on your side.

🛠️ ewx-cli​

• Detailed errors with --debug: ewx-cli now displays detailed error information when requesting a non-existent entity with the --debug flag enabled.
• CLI error exit codes: ewx-cli commands that fail with a platform API error now exit with a non-zero status code, so scripts and pipelines can detect the failure. Connection failures now show a clean error message instead of a Python traceback.
• Upload history fixes: 'ewx-cli config upload-history inspect' and 'revert' no longer crash on upload caches, 'purge --days' now deletes caches older than the given number of days as documented, and a reverted upload cache can no longer be reverted a second time.
• Dry-run for configuration releases: 'ewx-cli config upload', 'config transfer', and 'config upload-history revert' now accept a '--dry-run' flag that resolves dependencies, checks permissions, and prints the full execution plan without changing anything, so a release plan can be reviewed before it is carried out.

26.08.01​

⚙️ Custom Business Logic​

• Channel name fallback: Channels created by flows or ingestion no longer default their name to the technical classifier name. When a channel has no datasource-specific name, charts, tables and the channel list now show the classifier display name, falling back to the technical name only when no display name is set.

🚀 Notable Backend Changes​

• Timeseries edit history: TimeseriesService.load can now return the datapoint edit history alongside the channel values (include_edits=True), in an additional edits column. This lets custom business logic detect which datapoints were manually corrected through the Console. It is off by default, so existing reads are unchanged.

26.08.02​

Platform Bug Fixes​

• API tag writes now reach BigQuery: Fixed an issue where tags created through the API v2 tags endpoint were stored on the datasource but never synced to BigQuery, so they could not be found when querying tag data (for example in analytics or exports). Tags written through the API are now propagated to BigQuery the same way as tags created by flows.

🖥️ UI Improvements​

• Market adapters list: All available market adapters now appear in the market adapter list and file ingestion screens, instead of only the first 20. This unblocks ingesting files whose market adapter was previously missing. (New console)

⚙️ Custom Business Logic​

• Partially written flow results: A flow result that cannot be written no longer reports the whole flow run as failed. The results that were written are kept and the rejected ones are reported against the affected datasource and flow, but any follow-up flow in a chain is held back, because the run did not produce everything it was supposed to.
• Duplicate emails and tasks on retry: When a flow run had to be retried, any email it sent was sent again and any task without its own identifier was created again. Both are now recognised as already done and skipped on the retry.
• Suppressed follow-up triggers: When flow results cannot be written, the follow-up triggers that were not published are now reported in an error audit event naming the flow configuration that did not start, instead of being dropped silently. Side effects unrelated to starting a flow, such as search indexing, still run.
• Flow result write failures: When flow results cannot be written, the error audit event now names the affected datasource, flow and file instead of only the namespace, so the failure is findable from the entities you filter on. Repeated failures are each reported instead of only the first one.

26.08.03​

🔐 API & Permissions​

• Google sign-in no longer blocked by the web application firewall: On environments with the Cloud Armor WAF enabled, signing in with Google returned a bare 403 Forbidden page after the redirect back from Google. The firewall's remote-file-inclusion rule was tripping on the Google URLs that the callback carries in its scope parameter, because the rule that allow-lists the OAuth callback did not match the path the console serves it on. Both callback paths are now allow-listed.

📡 Integration Component​

• Outbound SFTP delivery failures now raise tasks: A file that fails to reach an outbound SFTP server now always creates a task on the configured board, including for message types generated by processing rules that carry no message ID. Previously those failures were silently discarded — the task creation was rejected and the message acknowledged — so no task or alert was raised for an undelivered file.
• ACK URL lookup fix: Resolved an issue where acknowledgement-specific URLs configured in the message URL map were silently discarded during URL resolution, causing non-GLDPM acknowledgements to fall back to the default URL and be rejected by the counterparty with a version mismatch.

What's New ✨​

New console (NUI)​

With Beta features enabled, the new console gains a wave of improvements:

• New look and feel across the console, with refreshed page headers, tabs, and controls for a cleaner, more consistent experience.
• New landing page — a card-based home screen for quick access to the areas you use most.
• Datasource view — the datasource detail view (chart, data table, tags, and annotations) is now available in the new console.
• Task management is now available in the new console.
• Environment switcher (opt-in) — jump straight to another Energyworx instance and namespace you're signed in to, right from the top bar.
• New settings page — User Settings reorganised into clear sections (Interface, Datasource, Tasks, Credentials, Account), plus a "Show namespace IDs" toggle.
• Search parity with the original console — the same features, now in the new console.

Ordered key processing​

Files uploaded with the same ordering_key are now ingested one at a time, in the order the platform receives them, instead of in parallel — preventing race conditions when several files target the same datasource (for example, sequential corrections for the same meter). Give each independent stream its own key so unrelated files still process in parallel. Set the key as a path component when uploading to GCS — see Uploading files to Energyworx (GCS).

Process monitoring start time alerts​

Process monitoring can now alert on when a process starts. Configure a condition on the "Process Start Time" metric with an HH:MM threshold (e.g. alert if a process hasn't started by 09:00) — the alert fires as soon as that time passes with no activity, and holds other conditions until the process actually starts.

🖥️ UI Improvements​

• Automated access settings: OAuth2 client credentials now live in their own "Automated access" section, separate from personal Account settings — making it clear they're for API and automation.
• No more freezing after a new release: If the web UI was open in a tab when a new version shipped, some pages could silently stop responding. The UI now detects this and refreshes itself so navigation keeps working.
• Clearer Groups & Permissions: The table now hides deprecated and internal resources and shows a short description for each one, so it's clearer what each permission governs.
• Audit event search: Searching audit events with API v2 keywords (like 'flows' or 'datasources') no longer hits the wrong endpoint and fails.
• Cleaner channel-classifier editing: The create-only "Datasource type" helper (not stored, and absent from the classic console and API) no longer appears when editing a channel classifier, so it can't show a misleading value for units used by more than one datasource type. The attribute-group editor was also restyled to match the rest of the form.
• Faster datasource tag edits: Editing a single tag now sends only that tag to the API instead of re-submitting all tags on every save.
• Trigger schedule version history restored: The version history panel is available again, and restoring a previous version now applies to the correct record.
• Process monitoring alert links: Links in process-monitoring alert emails now open the correct configuration under Observability, scoped to the alert's namespace.
• Default group protection: Users can no longer be removed from the default group — the option is hidden in the UI and rejected by the API.
• Sign-in page links: The "Sign in with Google" screen now shows the correct support contact and links to the published General Conditions page (replacing a broken Terms of Service link).
• Namespace properties: Saving a namespace property now works even when the namespace has none set yet (previously the first save silently did nothing).
• Choose your default data view: A new "Default data view" setting opens datasource data as a chart or a table by default — per user, with a namespace-wide default admins can set.
• Filter files by date range: File Management now filters files by a creation date range, with separate start and end datetime inputs.
• Automatic tags on manual uploads: Files uploaded through File Management are now tagged "console" and "manual" automatically.
• Warning before running a disabled trigger: Clicking "Run now" on a disabled trigger now shows a confirmation first, instead of failing with an unexpected error.
• Map XML attributes: When mapping an XML file in a transformation configuration, element attributes (e.g. id on <meter id="123">) are now selectable as columns alongside element values — no need to use the API.
• File upload & ingestion fixes: Uploads no longer fail with a CORS error; files stay editable even after a market adapter is assigned (the read-only lock now applies only to automated files); "ingest after upload" reliably starts ingestion; and re-ingesting an already-uploaded file no longer errors with "File IDs are required".

⚙️ Custom Business Logic​

• Market adapter file splitting: The XML and JSON market adapters now support a split_process_level property, letting large files be split into smaller chunks before processing to avoid timeouts on very large inputs. The existing json_process_level property continues to work as before. Processing of large JSON files is also noticeably faster.
• Duplicate datasource fix: Fixed a rare race during platform scale-up that could process an ingested file twice, creating duplicate datasource tags and audit events.
• Datasource tag types preserved: A standard package update fixes a bug that caused datasource-type tag properties to lose their type (falling back to plain text); dependent rules now behave as expected.
• Channel unit type available in BigQuery: A channel's unit type is now stored on the datasources table in BigQuery, so it can be used in dashboards and advanced search.
• Standard validation annotation properties: Property keys inside annotations written by the standard validation rules are now consistently sentence-cased spaced strings (e.g. Min n of intervals, Previous read value, Maximum allowed value), replacing the previous mix of snake_case, CamelCase, and partially-capitalized labels.
• Audit events for user group updates now correctly include details when a condition (additional specification) is added, modified, or removed.
• Rule framework tag loading: Fixed an issue where load_tags_multi_datasource in the rule framework incorrectly returned channel properties alongside datasource tags.
• ewx-public testing utilities: Fixed the ewx-public package so it now includes the ewx_public.testing module, restoring the ability to write unit tests for custom flow rules as documented.

🔐 API & Permissions​

• Larger timeseries queries no longer fail: Wide timeseries queries over multi-year ranges now return their full result instead of failing with a "message too large" (HTTP 429) error — the API accepts responses up to 64 MiB (raised from 4 MiB).
• Long-running API requests no longer time out: Long requests such as wide timeseries queries — whether called directly or through the web UI — now have up to 5 minutes to complete instead of failing with a 502 after ~2–3 minutes.
• Correct validation message for Channel Classifiers: Validating a Channel Classifier's technical name now returns a Channel Classifier error instead of incorrectly referring to a "datasource classifier".
• Tag removal now sticks: Tags sent with removed: true via the API v2 tag service are now saved as removed instead of silently reverting to active.
• Tags are searchable immediately: Tags created through API v2 are indexed for search as soon as they're created, matching API v1 behaviour.
• Datasource search with AND / OR: Searching datasources with boolean operators (e.g. A OR B) now returns the expected results instead of treating the operator as part of the search term.
• More reliable datasource listing: The datasource list endpoint now respects the requested page size and resumes correctly when paging through results. The filter and order_by options — previously accepted but ignored — were removed; use datasource search to filter by name or description.

📚 Documentation Updates​

• Documentation Structure Update: Documentation pages now have clean, stable URLs and every section has an overview page with card-based navigation; old links and bookmarks keep working via redirects. Search results also show a short description for every page.
• GCS file upload guide clarified: Updated the "Uploading files to Energyworx" client integration guide — the Python example now passes the project ID to the storage client, and the curl example uploads the object name verbatim (the object name must not be URL-encoded).
• Rule framework batch tag loading: Added documentation for load_tags_multi_datasource, a rule framework method that retrieves tags for multiple datasources in a single batch call without loading full datasource objects.
• Data ingestion guide: The data ingestion documentation now describes all five ways to get data into the platform - file upload through the file manager, inbound REST, scheduled REST polling, MODBUS headend, and SOAP messaging - with guidance on choosing between them.
• Rule code examples: Updated the custom rule documentation examples to use the current ewx-public package (FlowRule and TransformRule), with the legacy imports still noted as supported.
• Chained flow documentation: The Flow Design guide now documents the bulk chain flow rule and adds a comparison of the chain, bulk chain, and conditional chain flow rules - covering how many datasources each targets, whether they run conditionally, and their decision tree support - so it is clearer when to use each.
• ewx-public migration guide: Expanded the legacy-to-ewx-public import mapping to cover enums, domain models, utility functions and the testing module, and updated the rule code examples across the docs to use the current ewx-public imports.

🚀 Notable Backend Changes​

• Sign-in no longer re-prompts for consent: Signing in with Google or Microsoft no longer shows the IdP consent/permissions screen on every sign-in. Consent is now requested only the first time (or when a new refresh token is genuinely needed); the platform keeps the upstream refresh token server-side (encrypted) and reuses it on later sign-ins.
• Latest datasource view fix: Datasources with no channel assigned are now correctly included in the latest datasource view.
• Datasource update fix: Updating a datasource with no field changes no longer triggers an unnecessary write, preventing a new datasource version from being created and preserving the original creation timestamp.
• Datasource ID case normalization: Files with datasource IDs in any casing are now correctly matched to existing datasources, preventing duplicate entries or processing failures caused by case mismatches between the submitted ID and the stored ID.
• Flow-created channels stay linked to their datasource: Data a flow writes (derived channels, annotations) now keeps the datasource's original ID, so it no longer gets split from the source datasource by a casing mismatch.

📡 Integration Component​

• Inbound message tag validation fix: Incoming messages that carry no tags (for example N10 MeasurementSeriesNotification messages) are no longer rejected with the ewx-02: Object missing required field tags validation error; the tags field is now correctly treated as optional and the messages are processed.
• SFTP no longer re-uploads processed files: The SFTP integration now archives a file once it has been collected and forwarded, so the same file is no longer picked up and re-uploaded on every polling cycle.
• GLDPM acknowledgement completeness fix: Acknowledgements returned for incoming GLDPM messages now include the mandatory Reason/text element (alongside Reason/code) and the received_MarketDocument.type element, so market parties can submit revised GLDPM messages that were previously blocked.

🛠️ ewx-cli​

• Namespace display names: ewx-cli commands now accept a namespace display name (e.g. --namespace integration/automated_testing.dev) in addition to the underlying namespace ID, resolving the name against the namespaces visible to your authenticated user. When a name is ambiguous, an interactive session prompts you to pick the ID; non-interactive runs error out and list the candidates.
• Only require relevant permissions: ewx-cli config commands now check only the permissions for the resources you act on (viewer to read, owner to write) instead of requiring every config-management permission, and --no-deps no longer reads or requires permissions for unrelated dependencies.
• Automatic retry on transient errors: ewx-cli now retries requests that hit transient server-side failures (502/503/504 responses, or a connection reset or timeout before the request reaches the backend) with exponential backoff, so a momentary platform hiccup no longer fails a command outright.
• Process monitoring config transfer: Process monitoring configurations can now be downloaded and copied between namespaces using ewx-cli, removing the need to recreate them manually in the UI when promoting environments or onboarding new namespaces.

26.07.01​

📡 Integration Component​

• Inbound SFTP duplicate delivery fix: Successfully processed files are no longer removed from the Redis deduplication set immediately after archiving. Previously this caused duplicate Pub/Sub deliveries arriving within seconds to attempt to re-download an already-archived file, fail with a "file not found" error, and land in the dead-letter queue. Files now remain in the set for the full one-hour TTL so duplicate deliveries are recognised as already processed and skipped cleanly.

26.07.02​

🖥️ UI Improvements​

• The datasource search query and filters are now retained when navigating to a result's Details or Flows and back.

🔐 API & Permissions​

• Channel classifier consistency: Channel classifier attributes now behave consistently across API v1 and v2. The technical name is immutable, classifier ids may contain the '+' character, and the per-datasource channel name is optional.
• Channel labels: Charts, tables and the channel list now label a channel by its datasource-specific channel name when set, falling back to the classifier display name and then the technical name.
• Fixed access for users with mixed-case email addresses: Users whose identity provider (e.g. Microsoft Entra) returns their email with capital letters were incorrectly denied access to dashboards and other platform features. Authorization now matches the stored account regardless of email casing.

🚀 Notable Backend Changes​

• Datasource search now correctly returns results when filtering by classifier for newly-added datasources.
• Scheduled flow triggers can select datasources by query: the flow trigger schedule now accepts a datasource_ids_query (a BigQuery SELECT whose first column is a datasource id) as an alternative to a static datasource_ids list, resolved at execution time so the flow runs on whatever datasources currently match — letting a scheduled flow target a dynamic, self-maintaining set (e.g. every datasource whose name starts with a given prefix) with no id list to maintain.

📡 Integration Component​

• ACK URL lookup fix: Resolved an issue where acknowledgement-specific URLs configured in the message URL map were silently discarded during URL resolution, causing non-GLDPM acknowledgements to fall back to the default URL and be rejected by the counterparty with a version mismatch.

26.07.03​

⚙️ Custom Business Logic​

• Rule unit-test helpers restored (standard_test_utils): The ewx_public.testing module now includes the synthetic-timeseries generators — create_monotonic_series, create_sinusoidal_series, create_seasonal_series, create_register_channels and related helpers — available via from ewx_public.testing import standard_test_utils as stu. These restore the standard_test_utils generators that were previously only shipped with the legacy energyworx_public package, so custom-rule unit tests can build test series without it.
• Fixed a bug that prevented custom rules and scripts from importing the standard-packages library directly.

26.07.04​

⚙️ Custom Business Logic​

• Custom rules can be constructed and unit-tested directly (FlowRule): ewx_public.flow_rule.FlowRule now supports direct construction with keyword arguments, and add_tags records tag mutations onto tag_mutations (keyed by datasource id) exactly as the platform does — so custom-rule tests can build a rule and assert on the tags it produced without a mock.
• RuleTestCase and datasource mocks restored (standard_test_utils): ewx_public.testing.standard_test_utils is now a full drop-in for the legacy std-grid module — adding the RuleTestCase base class, mock_datasource_with_tags, and the remaining channel/timeslice helpers — so existing custom-rule test suites migrate with only the import line changing.
• Tag.to_dict() / TagProperty.to_dict() restored: Tag and TagProperty in ewx_public can again be serialized back to dictionaries and round-trip through from_dict(), with value_type emitted as a string — letting custom-rule tests assert tag mutations against expected dicts.

🚀 Notable Backend Changes​

• Datasource unit_type backfill fix: The 26.07 BigQuery datasource migration now backfills channels.unit_type on existing datasource version rows (preserving channel order), not only on newly-written rows. Previously the column was added but left empty, which blocked upgrading existing clients.

26.07.05​

Platform Bug Fixes​

• API tag writes now reach BigQuery: Fixed an issue where tags created through the API v2 tags endpoint were stored on the datasource but never synced to BigQuery, so they could not be found when querying tag data (for example in analytics or exports). Tags written through the API are now propagated to BigQuery the same way as tags created by flows.

What's New ✨​

Platform version in the new console​

You can now view the platform version directly from the console. This is available in the new console only — the version is shown at the bottom of the left-side menu, and clicking it takes you straight to the matching release notes. Please note that it may take a few hours after a deployment for the version to update.

ewx-cli updates​

This release brings a set of improvements to the ewx-cli:

• Auto ingestion fix: ewx-cli config upload --namespace-property market_adapters now merges new entries with the existing dict on the destination namespace instead of overwriting it, and prompts before replacing any colliding keys.
• Dashboard group transfer: ewx-cli config dashboards and ewx-cli config upload_dashboards now accept a --group option that fans the dashboard out to every user in a permission group on the destination namespace.
• Unchanged rules: ewx-cli config upload now labels rules whose display name, description, code, and parameters match the destination as (Unchanged) instead of (Update), and skips the no-op API call.

🖥️ UI Improvements​

• Chart timezone change: Switching the chart timezone on the datasource detail view no longer triggers a redundant timeseries refetch, and the localized chart and datapoint table now rebuild correctly on change.
• Namespace properties admin-only: The Namespace properties page in the old console is now restricted to admins, the entry button no longer appears for other users, and saving only patches the defaults key via the admin API so unrelated namespace properties can no longer be accidentally lost when the page fails to load.
• Sticky actions column in Firefox: Fixed an issue where action buttons in data tables stayed off-screen in Firefox until users scrolled horizontally. They now stick to the right edge alongside the column header, matching the behaviour in Chrome.
• Created-on filter for file management: The "Created on" date filter on the file management page now correctly returns matching files. It was previously returning empty after a recent API change.
• API documentation session refresh: The new console's API Documentation page now refreshes the platform session and transparently retries the request when a "Try it out" call hits an expired session, so the same Execute click returns the real response. Previously the page kept returning unauthorised until a manual reload.
• Profiling filter reload fix: Filter dropdown values in the Profiling view now reload automatically when the time range is switched while a filter is open, instead of remaining empty until the filter is closed and reopened.
• Friendly server-error dialog: When a request to the backend fails with a server-side error, the new console now opens a dialog explaining that the error is on our side and offers a one-click button to copy a debug block (timestamp, request ID, trace IDs, app version, namespace) that users can paste into a support message so we can locate the matching backend logs immediately.
• Fix for German language: Fixed an issue that prevented some German translations from being displayed correctly.

⚙️ Custom Business Logic​

• Process monitoring timezone: Process monitoring configurations now support a configurable timezone (IANA format, defaults to UTC) that controls when alerts are evaluated and when historical metric snapshots are taken. The timezone setting has been consolidated to the configuration level, replacing the previous per-alert timezone.
• Batch datasource and tag loading in rules: Rule code can now call self.load_multi_datasources(ids) to load multiple datasources in a single batch call, and self.load_tags_multi_datasource(ids) to fetch tags for multiple datasources without loading the full datasource objects.
• JSON market adapter performance: Improved the performance of the JSON market adapter so that it can now handle larger files without timing out during ingestion.

🔐 API & Permissions​

• Classifier ID validation: The API now rejects channel and datasource classifier IDs containing spaces, dots, or other special characters with a validation error. Only alphanumeric characters, hyphens, and underscores are allowed.
• Client credentials user attribution: User email is now correctly recorded in datasource audit logs and access token subjects when authenticating via client credentials, replacing the opaque client ID that was previously used.
• Datasource search filter fix: Datasource search now returns results when the search filter has leading or trailing spaces.
• Empty static value validation: Creating or updating a transformation configuration with an empty staticValue in any datasource mapping field is no longer allowed. The API now returns a 400 error, and the old console form blocks saving when a static value field is cleared during an update.
• IAM group API change: The userIds field has been removed from group responses and replaced with userCount. This also fixes inflated member counts that occurred when users were removed directly from the system without going through the IAM service.
• Default group deletion fix: Fixed an issue where the default system permission groups could be deleted via the API or UI, which would lock all users in the namespace out of every resource.

📚 Documentation Updates​

• Search help cleanup: Updated the search help sheet in both consoles to match what the backend actually supports — removed the outdated "Search suggestions" and "Multi-value queries" blocks, dropped the unsupported NOT and > / >= operator examples, and corrected the AND/OR precedence note (OR binds tighter than AND).
• Shared package versions: Documented that customer-facing package versions (ewx-cli, ewx-public, pytest-bdd-ewx) follow the platform release version (e.g. platform 26.03 ships package version 26.3.0), and corrected the stale install example on the ewx-cli page.
• ewx-public install docs: Corrected version references in the ewx-public package installation guide from 26.2.0 to 26.3.0.
• How to upload files to GCS: Added a guide on how to upload files to a GCS bucket, describing the process for multiple scenarios.
• Inbound integration service docs: Updated the inbound REST integration service documentation to use the correct per-environment endpoint URL format, removing outdated references.

🚀 Notable Backend Changes​

• ⚠️ In-house identity provider: We have migrated the platform's authentication from Google Firebase to an in-house identity solution. This is a behind-the-scenes change to how sign-in is handled and should not impact most users. However, if you sign in with Microsoft or SAML, additional configuration steps may be required.
• Timeseries aggregation timestamp fix: Fixed an issue where aggregated timeseries events had timestamps in seconds instead of milliseconds, causing datasources to be stored with incorrect timestamps.
• Large CSV ingestion fix: CSV ingestions that resolve more than 500 datasources per payload now succeed reliably; the datasource lookup is fanned out to Bigtable in batches instead of a single request that could time out. Payloads referencing more than 1,000 datasources also emit a warning audit event suggesting the user set datasource_id_columns on the market adapter or split the input file.
• Enhanced security hardening — Cloud Armor WAF: The platform's existing security infrastructure can now be extended with per-environment Cloud Armor WAF policies across all public-facing gateway and EnerGPT services. This adds an extra layer of protection against OWASP Top 10 attack patterns, automated scanner detection, and login brute-force attempts — complementing the existing network-level and application-level controls already in place.
• Split column fix: Fixed a bug that prevented the split column function in transformation configurations from working in some namespaces. This functionality now works correctly out of the box, upon namespace creation.

26.06.01​

⚙️ Custom Business Logic​

• Market adapter audit events: Datasources created or updated through the market adapter v2 now correctly emit audit events, ensuring a complete audit trail for all datasource changes.

📡 Integration Component​

• GLDPM acknowledgement matching fix: Resolved a bug where outbound GLDPM messages were not being correctly matched to their returning acknowledgements, causing the integration to be unable to confirm whether a GLDPM submission was accepted or rejected by the counterparty.

26.06.02​

⚙️ Custom Business Logic​

• Large flow output files: Fixed issue where additional output files around 10MB or larger would fail the flow. Now larger file sizes are supported by the additional output functionality.

📡 Integration Component​

• Large custom Pub/Sub payloads: Offloaded large payloads published via publish_to_custom_pubsub_topic now retain their original message attributes when delivered to EnergyWorx integration services, so downstream consumers receive the same metadata as for non-offloaded messages.

26.06.03​

• Cloud NAT static IP allocation: Environments that require stable outbound IP addresses (e.g. for client-side firewall whitelisting) can now be configured with reserved static external IPs for Cloud NAT, replacing automatic IP allocation. This prevents unwhitelisted IPs from being assigned during high-load events or deployments.

Platform Bug Fixes​

• Flow chaining fix: Fixed an issue where bulk chain flows could silently fail to trigger downstream flows due to incorrect Datastore key type resolution.

📡 Integration Component​

• Inbound acknowledgement routing fix: Acknowledgements for incoming SOAP messages are now sent to the correct acknowledgement endpoint rather than the notification endpoint, preventing counterparties from rejecting them with a validation error and leaving messages stuck in a retry loop.

26.06.04​

ewx-public testing utilities: Fixed the ewx-public package so it now includes the ewx_public.testing module, restoring the ability to write unit tests for custom flow rules as documented.

📡 Integration Component​

• SFTP no longer re-uploads processed files: The SFTP integration now archives a file once it has been collected and forwarded, so the same file is no longer picked up and re-uploaded on every polling cycle.
• Inbound SFTP duplicate delivery fix: Successfully processed files are no longer removed from the Redis deduplication set immediately after archiving. Previously this caused duplicate Pub/Sub deliveries arriving within seconds to attempt to re-download an already-archived file, fail with a "file not found" error, and land in the dead-letter queue. Files now remain in the set for the full one-hour TTL so duplicate deliveries are recognised as already processed and skipped cleanly.

26.06.05​

• Staging/production sign-in fix: Being signed in to a production environment no longer logs you out of its staging environment (and vice versa) when the staging site runs on a subdomain of the production site. The two sessions are now kept completely separate and no longer interfere. (EWXDEV-16488)

📡 Integration Component​

• Outbound SFTP delivery failures now raise tasks: A file that fails to reach an outbound SFTP server now always creates a task on the configured board, including for message types generated by processing rules that carry no message ID. Previously those failures were silently discarded — the task creation was rejected and the message acknowledged — so no task or alert was raised for an undelivered file.
• ACK URL lookup fix: Resolved an issue where acknowledgement-specific URLs configured in the message URL map were silently discarded during URL resolution, causing non-GLDPM acknowledgements to fall back to the default URL and be rejected by the counterparty with a version mismatch.

26.06.06​

⚙️ Custom Business Logic​

• Rule unit-test helpers restored (standard_test_utils): The ewx_public.testing module now includes the synthetic-timeseries generators — create_monotonic_series, create_sinusoidal_series, create_seasonal_series, create_register_channels and related helpers — available via from ewx_public.testing import standard_test_utils as stu. These restore the standard_test_utils generators that were previously only shipped with the legacy energyworx_public package, so custom-rule unit tests can build test series without it.
• Fixed a bug that prevented custom rules and scripts from importing the standard-packages library directly.

26.06.07​

Platform Bug Fixes​

• API tag writes now reach BigQuery: Fixed an issue where tags created through the API v2 tags endpoint were stored on the datasource but never synced to BigQuery, so they could not be found when querying tag data (for example in analytics or exports). Tags written through the API are now propagated to BigQuery the same way as tags created by flows.

What's New ✨​

Trigger schedules from Flows​

It is now possible to start trigger schedules from a flow, via the newly added function execute_trigger_schedule. For more details please check out our documentation here.

New UI pages available in Beta​

Additional pages are now available in the new console for Beta users:

• File Management, Market Adapters, and Transformation Configurations
• Channel Classifiers, Datasource Classifiers, and Trigger Schedules

Notable improvements that ship with these pages:

• Auto-ingestion property on Market Adapters: You can now configure the market_adapters property for auto ingestion directly from the Market Adapter page in the new console.
• Load a file from a Transformation Configuration: You can now navigate from a Transformation Configuration to a file uploaded in File Management. Previously, navigation was only possible in the opposite direction (file → Transformation Configuration).
• SQL formatting for Trigger Schedule queries: Trigger Schedule queries are now displayed with SQL syntax formatting, making them easier to read and review.

⚠️ MDM — Datasource and Flow API V2 services merged​

The Datasource and Flow API V2 services have been merged into a single MDM (Meter Data Management) service.

• URL change:
  • Before: datasource.<project>.ewxapis.com / flow.<project>.ewxapis.com
  • After: mdm.<project>.ewxapis.com
• API version change:
  • Before: /v1alpha1/
  • After: /v1/
• What to update: Replace datasource or flow in the hostname with mdm, and replace /v1alpha1/ with /v1/ in all request paths.
• Affected endpoints: Datasources, Datasource Classifiers, Channel Classifiers, Channel Families, Computed Flags, Flow Designs, Tags, Timeseries, Profiling.
• Not affected: Identity, Process Monitoring, and Market Adapter services remain on their existing hostnames with /v1alpha1/.
• Increased downtime after deployment: The MDM service will have a slightly increased downtime after deployment, meaning the New UI and API will not be available for around 30–60 minutes after deployment finishes.

⚠️ Deprecation Notice​

• Channel Flows: Channel Flows are deprecated as of 26.05 and will be removed in release 26.11. All Channel Flows must be migrated to Datasource Flows before that release. Please reach out through our Service Desk if you have any questions regarding this migration.

🖥️ UI Improvements​

• Project documentation (Preview): You can now add project-specific documentation that lets you surface implementation details to business users. This is possible by uploading a markdown file via API or ewx-cli. More information can be found here. The feature ships in preview; shape and lifecycle may still change.
• Profiling filters fix: Fixed the issue which caused profiling filters to disappear.
• File Management search: Now, the filter by day respects the user's local timezone and searches for files within that day in the user's local timezone.
• User settings respected on Datasource Detail page: Default date range is now applied correctly on the Datasource Detail page; from/to dates are pre-filled when opening a datasource. All time units (hours, days, months, years) are now supported (previously a configured "3 months back" was being interpreted as "3 years back"). The Audit Events page also reflects current settings — caching that previously required manual browser-storage clearing has been removed.
• Date picker cursor fix: Fixed an issue in the new console which caused the cursor to jump to the end of the input when changing the date in the datasource view.
• Chart zoom on large datasets: Fixed an issue that prevented zooming in on the chart when a lot of data was loaded (e.g. 1 year of 15-min data across 4 channels).
• KB / Service Desk duplicate tabs: Fixed an issue which caused the knowledge base to open duplicate tabs when opening it in a new tab via right click or cmd+click.
• Tag property keys are now read-only: Fixed an issue in the new console where the property key of an existing tag was editable; existing keys are now read-only and adding a property with a duplicate key is blocked.
• Sign out redirection with beta features: Fixed an issue which prevented correct redirection when signing out from the new console with beta features enabled.
• New console sign in page background: Added a new background to the new console sign in page.
• Datapoints clickable in "Show flags" mode: Fixed an issue where datapoints with flags could not be clicked in "Show flags" mode on the datasource chart, preventing the datapoint info panel from opening.
• "Last edited" field accuracy: Fixed an issue when retrieving timeseries edits that occurred when the same timestamp had edits for different channels with different versions.
• First-time login fix: Fixed an issue that prevented newly whitelisted accounts from logging in to the new console for the first time.
• Adblocker info message on Process Monitoring page: Added a message to Process Monitoring when there is an error advising the user to check for adblocker usage.
• German language support: Added support for the German language in the platform.

⚙️ Custom Business Logic​

• Timezone comparison fixes in rule framework: Resolved timezone-naive vs offset-aware datetime comparison errors in two code paths: filtering timeslice groups by start or end, and get_active_tag_versions when tag valid_from is timezone-naive. Timezone-naive objects are now converted to timezone-aware before comparison.
• Chain rules created upon namespace creation: The conditional_chain_flow and bulk_chain_flow rules are now created automatically when a new namespace is added. This was previously only true for the chain_flow.
• Tag validation: Added validation for tag property values across two paths: when stored via the rule framework (raising a ValueError that the user must handle, otherwise the flow is canceled and an audit event generated) and when applied during datasource creation/update on file ingestion (so failures due to transformation configuration type mismatches now generate proper audit events).
• ML prediction restored for FlowRule: The get_prediction() and promote_model() methods are available again on FlowRule.

🔐 API & Permissions​

• File storage datetime search: The file storage API now accepts complete datetime values in UTC, searching within a 24-hour range after the provided timestamp. If the user wants files in their local timezone, they need to convert to UTC beforehand.
• Client credentials creation: Fixed an issue where users could only create client credentials when they were whitelisted to the energyworx.org namespace. Any whitelisted user can now create client credentials for themselves.
• Datapoint versions on continuous flows: Fixed an issue that prevented the creation of a new datapoint version when editing a datapoint via API v2 on a continuous flow.
• Datasource fields after Flow runs via APIv2: Fixed an issue that was causing the datasource to be updated with wrong information when triggering a Flow through APIv2.
• Editable DPA enforcement on NUI / API v2: Fixed an issue that was allowing edit of DPAs even when the DPA was explicitly set to not be editable.
• Datasource IDs accept dashes: Datasource IDs may now contain dashes (-).

🛠️ ewx-cli​

• Dashboard transfer auto-remap: When transferring or uploading dashboards, the ewx-cli now automatically remaps flow IDs in dashboard string replacements, by matching flow names between source and destination. More information can be found here.
• Transactional config uploads with rollback: config upload is now transactional. If any step fails, the CLI automatically rolls back every configuration it had already applied, restoring the destination namespace to its pre-upload state and exiting non-zero with the original platform error. Behaviour on a fully successful upload is unchanged.
• dashboards fix: Fixed a crash in config dashboards that caused an AttributeError: 'tuple' object has no attribute 'iter_all_pages'. The command now correctly unpacks the client tuple returned by create_client().

📚 Documentation Updates​

• APIv1 documentation — clearer button label: The button in the APIv1 documentation now reads Execute instead of Try it out, making it explicit that the button executes the command.
• Customer onboarding & ewx-public testing framework: Added in-depth documentation about how to work with the ewx-public testing framework for developing flow and transform rules.

🚀 Notable Backend Changes​

• Python dependency security update: Updated urllib3 across all platform services to address a known vulnerability.
• Base image security updates: Updated container base images across all platform services to resolve HIGH and CRITICAL OS-level vulnerabilities identified by security scanning.
• Gateway security updates: Upgraded key dependencies in the API gateway to patch three known vulnerabilities (CVE-2025-30204, CVE-2025-46569, CVE-2025-22868).
• Infrastructure security hardening: Resolved multiple GCP Security Command Center findings by tightening network access controls and Terraform resource configurations on the platform infrastructure.
• Improved reliability of data processing pipeline: Improved how the platform handles recovery when a processing worker is unexpectedly interrupted (for example, during a node restart or scaling event). What you'll notice:
  • More consistent end-to-end processing latency, especially during high-volume ingestion.
  • Fewer cases where a single file appears "stuck" while others around it complete normally.
• Trace ID propagation: Trace IDs are now propagated through all processing and API components (Python and Go), making it easier to track messages through the entire system.
• User & namespace context in API logs: API request logs now include user and namespace information on all log entries, enabling easier filtering and tracing of individual user requests.

📡 Integration Component​

• SEL meter aggregate values fix: Fixed a bug in the modbus service so that the current data is only calculated once and reused for each LDP read retrieval, addressing insufficient meter reads on umic_com_001.
• Outbound acknowledgement tags fix: Tags on acknowledgement files uploaded to GCS were incorrectly populated from the original inbound message fields (senderId, creationTimestamp, messageId). The fields are now overridden with the ACK's own metadata before forwarding to the file service.

26.05.01​

• Audit Events schema compatibility: The writers service now accepts both the current and legacy Avro schemas for Audit Events, preventing write failures during the migration period until the legacy schema is fully retired.
• Search indexing for market adapter datasources: Fixed a bug where datasources created or updated by pluggable market adapters were never published to the search index.
• pyarrow in prepare-payload image: Added an explicit import pyarrow in the market-adapter loader so Pants' import inference keeps pyarrow in the prepare-payload build artifact and can be used in the Market-adapter codeBlobs.

26.05.02​

Platform Bug Fixes​

• get_active_tag_versions fix: Fixed a timezone comparison error. The function now ensures all returned tags utilize timezone-aware "valid from" timestamps.

26.05.03​

• SOAP list URL routing fix: Fixed an issue where SOAP integrations configured with a dedicated list URL in the URL map would incorrectly route requests to the default URL instead.
• SOAP MetaData header fix: Resolved an issue where SOAP requests for the MetaData message type incorrectly included a Content-Type header, causing those requests to fail.

Security Improvements​

* Enhanced security hardening — Cloud Armor WAF: The platform's existing security infrastructure can now be extended with per-environment Cloud Armor WAF policies across all public-facing gateway and EnerGPT services. This adds an extra layer of protection against OWASP Top 10 attack patterns, automated scanner detection, and login brute-force attempts — complementing the existing network-level and application-level controls already in place.

26.05.04​

• Namespace properties admin-only: The Namespace properties page in the old console is now restricted to admins, the entry button no longer appears for other users, and saving only patches the defaults key via the admin API so unrelated namespace properties can no longer be accidentally lost when the page fails to load.

26.05.05​

• Flow chaining fix: Fixed an issue where bulk chain flows could silently fail to trigger downstream flows due to incorrect Datastore key type resolution.

26.05.06​

ewx-public testing utilities: Fixed the ewx-public package so it now includes the ewx_public.testing module, restoring the ability to write unit tests for custom flow rules as documented.

• Datasource OR/AND search fix: Searching for multiple datasource IDs using boolean operators (for example, A OR B) now returns the expected results in the OUI datasource search bar. Note: mixed operators (e.g. A OR B AND C) are not yet fully supported and will be improved in a future release.

26.05.07​

⚙️ Custom Business Logic​

• Rule unit-test helpers restored (standard_test_utils): The ewx_public.testing module now includes the synthetic-timeseries generators — create_monotonic_series, create_sinusoidal_series, create_seasonal_series, create_register_channels and related helpers — available via from ewx_public.testing import standard_test_utils as stu. These restore the standard_test_utils generators that were previously only shipped with the legacy energyworx_public package, so custom-rule unit tests can build test series without it.
• Fixed a bug that prevented custom rules and scripts from importing the standard-packages library directly.

26.05.08​

Platform Bug Fixes​

• API tag writes now reach BigQuery: Fixed an issue where tags created through the API v2 tags endpoint were stored on the datasource but never synced to BigQuery, so they could not be found when querying tag data (for example in analytics or exports). Tags written through the API are now propagated to BigQuery the same way as tags created by flows.

What's New ✨​

ewx-public package:​

Introduced a new package called ewx-public that contains all models and service stubs needed to develop and test rules outside the Energyworx platform. To learn more, check out our documentation here.

Updated Search​

The datasource search service has been modernized, delivering faster searches and improved reliability. As a bonus, full re-syncs now complete in minutes instead of hours.

⚠️ Deprecation Notice​

• Legacy API URL: The api.<short_env_id>.energyworx.net URL format is officially deprecated and will be deactivated in release 26.10. All integrations must migrate to the new ah.<full_env_id>.ewxapis.com format. Please reach out through our Service Desk if you have any questions regarding this change.

🖥️ UI Improvements​

• New sign in page: The platform's sign in page has been updated to only show the default sign-in option. This is to avoid confusion between which provider to use - other options are available by clicking 'other sign-in options'
• Simplified login: Removed the unsupported sign-up option to create a cleaner interface; this update has no impact on existing user functionality.
• Expanded list views: Added support for loading 250 and 500 items at a time across all relevant list views in the platform.
• Future tag deletion: Fixed a bug which prevented tags with a valid_from date in the future to be successfully deleted directly from the console.
• Multiple transform rules: Resolved a UI limitation to allow multiple transform rules to be added to a single field in a Transformation Configuration. This was previously only possible via API.
• Tag date formatting: Fixed a validation issue in the new console that caused invalid formatting when updating a tag's valid_from field.
• Timezones in Advanced Search: Resolved a bug in Advanced Search that prevented localized timezones from displaying correctly in the frontend.
• Annotation edit tracking: Fixed an issue where annotation edits were not correctly rendering in the "last edited" column within the new console.
• User status accuracy: Updated the namespace user list to correctly display disabled users as "Inactive" instead of "Active."
• First-time new console login fix: Fixed an issue that prevented newly whitelisted users from logging into the new console for the first time.
• Audit Event performance: Optimized query performance for Audit Events within the new and original console to resolve loading delays in high-volume environments.
• Audit Event search fix: Resolved a bug that prevented the system from finding audit events when search requests included single quotes.
• Rule Profiler filtering: Fixed an issue where the Flow ID and Sequence ID filters were non-functional within the Rule Profiler interface as well as an issue which caused filters to not appear.

⚙️ Custom Business Logic​

• Transform Rule error handling: Enhanced generic exception error messages in Transform Rules to provide clearer context for troubleshooting.

• Merged Column fix: Fixed bug which prevented 'Merged Column' function in Transformation Configurations from working.

• Tag type fix: Fixed issue which prevented tags of type 'link' and 'date' to be configured in a Transformation Configuration

• Rule validation accuracy: Fixed a bug that returned an incorrect error message when attempting to save a rule where the technical name and class name did not match.

• Tag retrieval fix: Resolved a "timezone-naive vs offset-aware" comparison error in the rule framework, within the get_tag method to ensure consistent datetime handling.

• Pytestbdd-ewx fix: Resolved a pytest-bdd-ewx issue encountered when checking tags with multiple versions that share the same valid_from timestamp.

• Market Adapter structural update: Resolved a naming conflict bug by separating Market Adapter implementations from their configurations, preventing 'NoneType' errors.

🔐 API & Permissions​

• Datapoint edit history fix: Resolved an APIv2 bug where the "previous value" was not correctly returned after editing a datapoint that held a value of 0.

• API documentation filtering: Fixed a bug in the APIv2 documentation interface that prevented users from successfully finding channels when applying filters.

• Latest Datapoint filter fix: Resolved an issue where the channel family filter was not functioning correctly within the Latest Datapoint API.

• Classifier case-sensitivity normalization: Fixed an issue that allowed Datasources to be created with lowercase classifiers by implementing automatic conversion to uppercase; additionally, both Datasource and Channel classifiers are now treated as case-insensitive within the processing pipeline.

• Process Monitoring access control: Introduced a dedicated Process Monitoring permission specifically for managing configuration access; users now also require permissions for flows and market adapters to fully utilize all monitoring functionalities.

📚 Documentation Updates​

• Enhanced documentation discovery: Release notes are now fully indexable and findable via the documentation search bar, with unique URLs assigned to each individual release note page.
• Pandas 2.0 migration guide: Added a comprehensive migration guide for transitioning from Pandas 1.x to 2.x; Pandas 1.x is now officially deprecated and will be upgraded to 2.x starting in version 26.x.
• Isolated Shareable Packages: Refactored the documentation structure to isolate "Shareable Packages" (such as pytest-bdd and ewx-cli) to improve navigation and clarity.
• Independent doc deployments: Decoupled the documentation from platform deployments, allowing for content updates to be published independently and more frequently.
• New technical guides: Published new documentation covering the usage of Decision Trees and updated the implementation guides for Flow Properties.
• Live Flags limitations: Improved the Live Flags documentation by adding specific details regarding condition types and their current limitations.

🚀 Notable Backend Changes​

• Pandas 2.x compatibility: The platform is now compatible with Pandas 2.x. Keep in mind that Pandas 1.x is currently still used, this is just for forward-compatibility.
• Duplicate channel fix: Fixed multiple bugs that caused duplicate entries in the latest_datasources table — both when creating a datasource via API with a channel and running a flow, and when adding channels to an existing datasource via API v1.
• Silent trigger failure now logged: Added an audit event when a trigger schedule fails due to a non-existent Market Adapter. Previously this failed silently.
• Datasource ID validation: Added validation for datasource IDs created through the pipeline to prevent invalid entries.
• Processing performance: Several small improvements to the data processing pipeline that should reduce processing overhead by approximately 20% in low-volume situations.
• PubSub optimization: Refined message size validation and lowered internal limits to ensure reliable delivery.
• Security scanning: Added Trivy for SAST (Static Application Security Testing) and SCA (Software Composition Analysis) to the platform's CI pipeline.

📡 Integration Component​

SFTP Integration Support​

The Integration Component now supports SFTP-based file integrations alongside the existing SOAP protocol. A unified scheduler architecture handles both protocols as a single cronjob, connecting to configured SFTP servers on schedule, retrieving files matching configured patterns, and forwarding them to filemanager for ingestion. Like SOAP, the SFTP service supports inbound and outbound flows, XSD validation, field mapping, and custom validation rules — all driven by namespace-specific configuration stored in the GCP bucket.

• DST timestamp correction: Resolved an issue with SEL meter timestamp generation during Daylight Saving Time (DST) transitions to ensure accurate data; updated the retrieval script logic to support specific start and end date ranges.

26.04.01​

• Improved reliability of data processing pipeline: We've made an improvement to how the platform handles recovery when a processing worker is unexpectedly interrupted (for example, during a node restart or scaling event).
• What you'll notice:
  • More consistent end-to-end processing latency, especially during high-volume ingestion.
  • Fewer cases where a single file appears "stuck" while others around it complete normally.

📡 Integration Component​

• Inbound file corruption fix: Resolved a bug in the SFTP inbound logic that caused the file delivered to EDM to differ from the original file on the server, resulting in files consistently failing at the TC step when ingested via EDM.

• Insufficient meter reads fix: Resolved an issue where the SEL data retriever was computing a new timestamp for each LDP during execution. In long-running executions this caused a drift between the saved start time and the actual retrieval start time of the first channel, leading to missed reads and incorrect aggregate values.

26.04.02​

📡 Integration Component​

• SFTP re-ingestion fix: Resolved a bug where an already-processed file on the SFTP server was re-uploaded on every scheduler cycle. Processed files are now archived under a unique timestamped name, preventing the same file from being picked up again on subsequent runs.
• GLDPM acknowledgement completeness fix: Resolved an issue where GLDPM acknowledgements were missing two mandatory fields — Reason/text and received_MarketDocument.type — which caused counterparties to reject revised GLDPM messages. Both fields are now correctly populated. Requires the corresponding config update per namespace: add the document_type field map and reason_texts entries to the message map file for the GLDPM message type
• DataSource tags validation fix: Fixed a crash where msgspec failed to decode a DataSource object when the tags field was absent from the payload by making the field optional with a default empty list.
• SFTP outbound namespace filtering: Outbound SFTP messages arriving for namespaces that are not configured for SFTP in the active environment are now acknowledged and ignored, preventing spurious processing errors.
• Outbound message broker timezone fix: Fixed a crash in the outbound message broker caused by a timezone mismatch when comparing message timestamps.
• Outbound message broker header fix: Fixed a tuple unpacking error in the outbound message broker when processing SOAP message header namespaces.

26.04.03​

• get_active_tag_versions fix: Fix timezone comparison error: resolves a TypeError when tags with timezone-naive valid_from are compared against timezone-aware datetimes.
• Legacy API URL routing fix: Resolved an issue with routing for the legacy API URL (api.<short_env>.energyworx.net).
• load_datasources_by_virtual_datasource_filter restored: Restored the rule framework method load_datasources_by_virtual_datasource_filter, which had been raising NotImplementedError since the FlowRule refactor. Rules using this method to resolve a virtual datasource to its underlying concrete datasources now work as before.

📡 Integration Component​

• SFTP Filenames fix: Outbound SFTP filenames now preserve the full timestamp — files with ISO timestamps in their name were being silently truncated on Windows-backed SFTP servers, causing files to overwrite each other. This is now resolved.
• SFTP Filenames Improvement: Outbound SFTP processor now honours a caller-supplied filename — when a filename is passed in the Pub/Sub message attributes, it is used as the upload filename on the SFTP server.

26.04.04​

Platform Bug Fixes​

• get_active_tag_versions fix: Fixed a timezone comparison error. The function now ensures all returned tags utilize timezone-aware "valid from" timestamps.
• Search indexing for market adapter datasources: Fixed a bug where datasources created or updated by pluggable market adapters were never published to the search index.

📡 Integration Component​

• GLDPM acknowledgement matching fix: Resolved a bug where outbound GLDPM messages were not being correctly matched to their returning acknowledgements, causing the integration to be unable to confirm whether a GLDPM submission was accepted or rejected by the counterparty.

26.04.05​

• Flow chaining fix: Fixed an issue where bulk chain flows could silently fail to trigger downstream flows due to incorrect Datastore key type resolution.

📡 Integration Component​

• SOAP list URL routing fix: Fixed an issue where SOAP integrations configured with a dedicated list URL in the URL map would incorrectly route requests to the default URL instead.
• SOAP MetaData header fix: Resolved an issue where SOAP requests for the MetaData message type incorrectly included a Content-Type header, causing those requests to fail.
• ACK URL lookup fix: Resolved an issue where acknowledgement-specific URLs configured in the message URL map were silently discarded during URL resolution, causing non-GLDPM acknowledgements to fall back to the default URL and be rejected by the counterparty with a version mismatch.

26.04.06​

Platform Bug Fixes​

• API tag writes now reach BigQuery: Fixed an issue where tags created through the API v2 tags endpoint were stored on the datasource but never synced to BigQuery, so they could not be found when querying tag data (for example in analytics or exports). Tags written through the API are now propagated to BigQuery the same way as tags created by flows.

What's New ✨​

Rule Profiling​

The new Profiler gives you visibility into the CPU performance of your processing rules and flow configurations. Using interactive flame graphs and time-based analysis, you can quickly identify which rules are most CPU-intensive, pinpoint bottlenecks within individual functions, and compare performance across datasources — making it easier to optimize your data processing workflows. To learn more, check out our documentation here.

Trigger schedule versioning​

It is now possible view the edit history of trigger schedules and perform rollbacks to previous configurations directly from the UI. ⚠️ Please note: older trigger schedules, created before this release, will show the user ID in the last edited by instead of the user's email. Once updated, the email will correctly show up. In any case, users with IAM permissions can convert user ids to user emails if necessary.

Market Adapter Framework​

Introduced several new functions to the Market Adapter framework, enabling users to directly create/retrieve Datasources, Tags, and Timeseries, as well as trigger flows directly from the adapt() method. On top of that, the following were updated:

• Domain object validation: Added strict validations for domain objects output from a Market Adapter, including ISO standard checks for Timezones and existence checks for channel classifiers.
• Annotation handling: Updated Market Adapter timeseries output to ensure annotations and sequence_id are handled properly during direct ingestion.
• Documentation: Updated documentation to reflect all these changes and make the market adapter framewor easy to use.

Deprecation Notice​

Later this year, we’ll be moving the platform from pandas version 1 to version 2. We currently expect to make this switch in release 26.09; while this date may shift, the upgrade will occur no earlier than 26.09.

Since some functions are deprecated in pandas 2.0, some rules might need refactoring to stay compatible. To make this as painless as possible, we’re doing the following

• We’ll provide best-practice documentation on upgrading and testing your rules for compatibility
• We’ll ensure that all rules in the standard package are updated by the time we upgrade

⚙️ Rule Framework​

• Local development maintenance: Refactored the packages necessary to develop and test rules outside the Energyworx platform to simplify and standardize them. This has no impact in existing packages and imports, and it is not yet avaible. It will be made available in the next release, through the GAR.

🖥️ UI Improvements​

• Graph accuracy fix: Resolved an issue where "NULL" values that had a DPA were incorrectly displayed as "0" in frontend graphs; these are now clearly distinguished.
• Last datapoint per channel: Added functionality to the new console which allows users to eaily see what is the most recent reading available for each channel classifier.
• Clickable chart legends: Restored the ability in the original console to plot or unplot lines in a chart by clicking directly on the channel classifier name in the legend.
• User edit traceability: Fixed a bug where edits made by users with "nan" in their email address appeared as "edited by no one"; this fix works retroactively for past edits.
• Flow view: Fixed a bug that caused some flows to not be retrieved when clicking in the Get Latest Flows button.
• Live flags: Fixed an issue that was preventing the first Enum option in the Live Flags to be picked when using the UI to set up live flags.
• Spanish support The platform is now also avilable in Spanish. You can change the language in the user settings if you wish to use it.

🔐 API & Permissions​

• Last datapoint per channel: Added a new endpoint to API V2 to retrieve the last available datapoint for a channel classifier.
• Least Privilege for Admins: Reduced the default power of the Admin resource to the bare minimum required for environment administration (global whitelisting and namespace management).
• Read-only default groups: All default permission groups are now read-only by default and cannot have their roles changed, ensuring core security settings remain intact.
• APIv2 Datasource validation: Fixed several validation issues in APIv2 to prevent the creation of datasources with invalid timezones, filters, or classifiers.
• Duplicate tag prevention: Updated APIv2 to detect and reject duplicate tags where only the valid_from attribute was changed to a future date, returning a 409 Conflict Error.
• Endpoint cleanup: Removed the redundant and non-functional PATCH datasource endpoint in APIv2; users should use the POST method for updates.
• ResouceInput in Flows: Fixed an API issue which caused flow runs to fail in some cases where a rule had a parameter of type ResourceInput

📚 Documentation Updates​

• Live Flags guide: Added a new page to the knowledge base covering the setup and visualization of Live Flags.
• Admin access details: Improved documentation regarding the new "Administrator" tab and how to manage restricted admin rights.
• Technical inheritance docs: Updated documentation for nested rules and multiple class inheritance.
• Namespace properties: Added new guides on how to update namespace properties.
• Tags updates via API: Updated documentation regarding how to update tags via the API

🚀 Notable Backend Changes​

• Tag version deduplication: Fixed a comparison method bug that was creating redundant tag versions for non-SCD tags even when properties remained unchanged.
• Pod autoscaler improvement: Fixed a memory limit issue in the ewx-pod-autoscaler that caused processes to get stuck.
• Collective trigger optimization: Significant performance improvements for collective triggers, ensuring faster processing of large message volumes (100k+).
• Memory leak resolution: Fixed a memory leak within the Datasource service in APIv2 that was identified during performance testing.
• Error handling on ewx-cli: Improved error handling in the ewx-cli, to ensure clear error messages are thrown where possible.
• Package versioning: Fixed an issue where certain packages in the GAR were not correctly versioned. Documentation has also been improved to be more detailed on the required steps to use said packages.

26.03.01​

Platform Bug Fixes​

• Improved Audit Event query performance: Optimized the Audit Event query in the new console to significantly improve loading times when retrieving large volumes of audit data.

Integration Component Fixes​

• SEL meter timestamp correction around DST transitions: Fixed an issue where SEL meter timestamp generation produced incorrect timestamps during Daylight Saving Time (DST) clock changes. The data retrieval script has also been updated to support fetching data for a specific date range (start and end date).

26.03.02​

Platform Bug Fixes​

• get_tag fix: Fixed an issue in the get_tag function which failed when tags were manually created in the platform.

26.03.03​

Platform Bug Fixes​

• PubSub Optimization: Refined message size validation and lowered internal limits to ensure reliable delivery.

26.03.04​

Platform Bug Fixes​

• Profiler Optimization: Fixed sequence ID and flow ID filter in the rule profiler
• Endpoints Optimization: Fixed issue that prevented valid certificates from being generated for one of the API url variants.

26.03.05​

Platform Bug Fixes​

• Trigger local time DST fix: Fixed an issue where triggers migrated from UTC to local time execution produced incorrect timestamps during DST transitions, causing data drift in subsequent data exports.
• New console account login fix: Resolved an authentication error that prevented newly whitelisted accounts from logging into the new console (NUI), which resulted in a 500 Internal Server Error.

26.03.06​

Platform Bug Fixes​

• Legacy API URL routing fix: Resolved an issue with routing for the legacy API URL (api.<short_env>.energyworx.net).

26.03.07​

Platform Bug Fixes​

• load_datasources_by_virtual_datasource_filter restored: Restored the rule framework method load_datasources_by_virtual_datasource_filter, which had been raising NotImplementedError since the FlowRule refactor. Rules using this method to resolve a virtual datasource to its underlying concrete datasources now work as before.

• get_active_tag_versions fix: Fix timezone comparison error: resolves a TypeError when tags with timezone-naive valid_from are compared against timezone-aware datetimes.

26.03.08​

Platform Bug Fixes​

• get_active_tag_versions fix: Fixed a timezone comparison error. The function now ensures all returned tags utilize timezone-aware "valid from" timestamps.

26.03.09​

• Flow chaining fix: Fixed an issue where bulk chain flows could silently fail to trigger downstream flows due to incorrect Datastore key type resolution.

26.03.10​

Platform Bug Fixes​

• Identity namespace access fix: Fixed an authorization issue where users not whitelisted in the energyworx.org namespace were unable to perform actions in the identity namespace.
• Store rule channel updates: The Store rule now persists new channels through the asynchronous datasource sink instead of performing synchronous writes during rule execution, improving reliability and consistency with the rest of the processing pipeline.

26.03.11​

Platform Bug Fixes​

• API tag writes now reach BigQuery: Fixed an issue where tags created through the API v2 tags endpoint were stored on the datasource but never synced to BigQuery, so they could not be found when querying tag data (for example in analytics or exports). Tags written through the API are now propagated to BigQuery the same way as tags created by flows.

What's New ✨​

Get_latest_datapoint Function Added In Rule Framework​

We've added a function (get_latest_datapoint) to the rule framework which allows you to get the last available datapoint for a channel directly, instead of repeatedly loading data to find it. To learn more, check out our documentation here.

Fixed case sensitive issue with timeseries data​

We've resolved an issue where Datasource IDs were treated as case-sensitive in the timeseries table, which previously caused data visibility issues when IDs didn't match exactly (e.g., 'DS_name' vs 'ds_name'). Datasource IDs in the timeseries table are now case-insensitive, aligning with all other platform tables.

⚠️ Important: This update includes a change to how data is stored in BigTable. A small data migration is required during deployment; this will be managed by our support team and is expected to have zero impact on environment downtime.

Disabled EnerGPT Preview​

EnerGPT preview has been disabled in the platform. The feature can be made available, at a cost, if you wish to enable it please contact our support team.

🖥️ UI​

• Permission condition restrictions: Updated the interface to correctly reflect that conditions are not supported for MLModels, Channel Families, and Live Flags permissions.

• Improved tag traceability: Fixed an attribution error across both the original and new consoles to ensure that created or updated tags correctly display the actual user in the created_by field instead of "System."

🛠 Error Handling​

• Rule inheritance: Resolved an issue that prevented audit events from being logged when a rule had an invalid parent class; an audit event is now triggered if a class attempts to inherit from multiple classes.

• Enhanced RuleResult validation: Improved audit event clarity when the return object in RuleResult is invalid by providing specific details on the cause of the validation failure.

🔐API and Permissions​

• User group removal sync: Fixed a synchronization issue ensuring all rights are correctly revoked and user details are updated immediately after a user is removed from a permission group.

• Cross-console permission parity: Resolved inconsistencies between consoles to ensure all permissions granted in the original console are correctly propagated to the new console.

• Append-only Datasource validation: Updated APIv1 to enforce "append-only" logic; any attempt to remove existing channels from a datasource will now return a 400 Bad Request to ensure data integrity and documentation alignment.

🏷️ DPAs​

• Improved documentation: Added comprehensive documentation and step-by-step instructions for creating Datapoint Attributes (DPAs) directly through the console.

• Timezone-aware DPA ingestion: Fixed a bug where Timestamp DPAs were incorrectly converted to naive objects; they now correctly remain timezone-aware during ingestion.

🚀Platform & Backend Improvements​

• Extended search capacity: Resolved a limitation in the search functionality that capped queries at 15 characters; users can now search using strings of any length.
• Advanced Search view restoration: Fixed an issue preventing the use of the v_channel_data_and_annotations_latest_versions_pretty view, which is now fully queryable in Advanced Search.
• Final deprecation of Transform Functions: Completed the removal of legacy Transform Functions (previously replaced by Transform Rules), ensuring they are no longer accessible within the platform.
• Fixed auth bug: Fixed an issue that was causing users being not authorized in some namespaces after using APIv2 or JWT tokens.
• Deployment Improvement: Included backend changes to reduce manual steps needed for deployment
• Improved Logging Added JSON-formatted logs to BigTable and BigQuery writers
• Record duplication cleanup: Fixed multiple backend logic errors that were causing duplicate and triplicate entries for the latest_datasource and datasource BigQuery tables.

26.02.01​

Bug Fixes​

• Fixed Frontend bug: Fixed an issue regarding the google analytics API, that was causing the Frontend Build to fail

26.02.02​

Bug Fixes​

• Fixed Tag Version bug: Fixed an issue which caused multiple duplicate tag versions to be created for tags that were not slowly changnig dimension

26.02.03​

Bug Fixes​

• Fixed documentation bug: Fixed an issue in the knowledge base which caused images and videos not to display

What's New ✨​

Run Trigger Schedules From The Console​

We've added a Run button to the trigger schedule details page. Now, you no longer need to run it from the API, but can start your test directly from the configuration page.

Processing Time Span Added As Alert Metric​

Process monitoring alerts can now be configured for the Processing Time Spam metric. This means it is now possible to set alerts to be sent off if a process takes longer than X hours to complete.

⚙️ Rule Framework​

• Improved rule parameter configuration: When adding a ResourceInput widget as a rule parameter, the default type is now a list. This means it is, by default, possible to add multiple resources (ie: multiple flows, datasources, etc). ⚠️Note: this means even if only one item is added, the rule framework will return a list. This does not impact existing rules, but all new rules created. It should be taken into account when inheriting rules which have a parameter of type ResourceInput

• Flexible rule inheritance: Enabled the use of custom rules as base classes, removing the requirement to inherit directly from AbstractRule or AbstractTransformRule. Please note: inheriting from multiple classes is not supported

🖥️ UI improvements​

• Versioned API documentation: Each api version now has its own dedicated documentation page, which is accessible from both consoles.

• Datasource View fix: Resolved a loading issue in the Datasource View that occurred when no source channels were present.

• Flow creation defaults: Updated the default setting for new flows to Continuous | Datasource to align with the most frequent use cases.

• Improved tag traceability Fixed issue where tags created from the console showed up as created by the System instead of the user who created it.

• User group sorting: Implemented numerical ordering by ID for user groups in the console.

• Faster search Added support for loading 250 and 500 results at a time for File Management, Search and Audit Events

🔌 API​

• BigQuery Sync Fix Fixed issue which prevented updates to datasources made through APIv1.5 to be reflected in BigQuery.

• Improved ingestion validation When a trigger of type bigquery-ingest is executed with the same reference date (and a previous one has been executed successfully) a Conflict Error is returned (HTTP Error Code 409) as opposed to a 200, as it was the case.

• Cross-console tag compatibility: Resolved a corruption issue to ensure tags can be seamlessly created and edited across both the original and new consoles.

🛠 Error Handling & Stability​

• Virtual Datasource Filtering: Improved handling for cases where a filter returns no datasources within the rule framework.

• Sending Emails: Added audit events for errors that were previously failing silently. For instance, when having duplicate emails on to_emails and bcc_emails lists.

• Timeseries Storage: Improved error messaging when attempting to use store_timeseries with an invalid channel_id.

• Nested Rules: When loading a nested rule fails, the error audit event will now show in the Datasource's detail page. The message was also improved to show the rule that is trying to use the rule that is failing to be loaded.

📚 Documentation Updates​

• Flow Properties: Updated documentation regarding allowed types in flow properties.

• CLI Resource Files: Improved the ewx-cli documentation for managing resource files.

• Permissions: Refined documentation explaining the whitelisting permissions logic.

🚀 Notable Backend Changes​

• ⚠️ Process Pools Implementation In processing, billing accounts are now replaced by process pools, which behave the same as before. Process pools will be used, from now on, to separate the workloads in which processing happens.

  Note: Billing accounts remain as they are in terms of user whitelisting and access, but no longer define how processing is ran.

• Processing Standardization: The rule framework now only receives standardized objects instead of legacy. This backend change cleans up the usage of legacy components in our processing, which will streamline and standardize the backend and make it easier to fix errors and develop new features in the future. (Note: This is a backend change with no impact on the end-user).

• Improved XML File Creation: Fix issue when trying to create XML response files. This issue was caused by the mechanism to remove duplicated elements and attributes.

• Fixed detector bug Fixed issue in Avro schema in the detectors writers

26.01.01​

Bug Fixes​

• Fixed auth bug: Fixed an issue that was causing users being not authorized in some namespaces after using APIv2 or JWT tokens.
• Deployment Improvement: Included backend changes to reduce manual steps needed for deployment

26.01.02​

Bug Fixes​

• Fixed Frontend bug: Fixed an issue regarding the google analytics API, that was causing the Frontend Build to fail

26.01.03​

Bug Fixes​

• API certificate fix: Fixed issue that prevented valid certificates from being generated for one of the API url variants.
• Fixed an issue with Trigger Schedules: Updated trigger logic to provide full support for daylight saving time (DST). Previously, triggers were locked to UTC offsets, which caused scheduled tasks to shift by one hour when regional time zones switched from DST to standard time. With this update, triggers now correctly respect regional time zone rules, ensuring that tasks run at the exact local time.

Send emails from the rule framework​

A send_email function has been added to the rule framework, which allows users to send emails from rules. To find out how, check out our knowledge base by searching "Sending Emails" or by navigating to Platform > 3- Data Processing > Custom Business Logic > Custom Rules > 7- Sending Emails.md

Trigger schedule supports JSON output​

Trigger schedules can now also output json files. To do that, you can add the property "format" and set it to "JSON". More details are available in the trigger schedule documentation

ewx-cli self-service authentication​

The ewx-cli tool has been updated such that the authentication no longer requires the help of the Service Desk team. Users are able to authenticate themselves, through the Settings page in the new console.

⚠️BREAKING CHANGE ⚠️

• Existing authentication with the cli will no longer work, when running the ewx-cli the next time you will be prompted to reauthenticate.
• Please note: The ewx-cli now supports using the subdomain from the platform URL to identify the project. While old project names are still supported, using the subdomain is the preferred method.
  • For example, if your platform URL is test-t.energyworx.net, you can use test-t/enrx_org_001.
  • Similarly, for test.t.energyworx.net, you can use test.t/enrx_org_001.

Notable Improvements & Fixes 🛠​

🖥️ UI improvements​

• Process Monitoring: Process Monitoring is now available through the original console and no longer needs Beta features enabled to be accessed.

• Boolean Display Consistency: Booleans now read as True/False instead of Yes/No in the original console. This means that a tag property that is a boolean will now correctly display True instead of Yes. (Note: this is only a front-end change and does not impact rule implementation).

• User-Friendly Task Configuration: Improved the task configuration page to show more user-friendly wording in the original console.

• Clearer Rule Saving Warnings: When saving a rule, the warning message now specifies which fields require updating the flow, making it clearer to users when this use case applies.

🐛 Error Handling​

• Improved Live Flag Validation Feedback: Corrected an issue where the system silently failed on saving invalid Live Flag configurations. The platform now implements proper validation feedback, ensuring that users receive a descriptive error message specifying the configuration conflict.

• Datasource Audit Events: Fixed issue which prevented an Audit Event from being logged when a user accessed a datasource in the new console. This corrects an issue with traceability and now allows audit events to track any user that accesses a datasource, regardless of which console they use.

• Improved DPA/Annotation Error Logs: Improved error log when saving DPA or annotations using store_timeseries. The message now specifies clearly what the issue is, making it easier for the user to correct the code.

⚙️ Platform & Backend Improvements​

• Flow Property Lists: It is now possible to pass lists as values within flow properties.

• Datasource Handling Standardization: Standardized datasource handling in the rule framework. This backend change cleans up the usage of legacy components in our processing, which will streamline and standardize the backend and make it easier to fix errors and develop new features in the future. (Note: This is a backend change with no impact on the end-user).

• FlowCancelException Fix: Fixed an issue which prevented the show_exception_as_warning parameter to work when imported from the energyworx module.

🔐 API & Permissions​

• Channel Family API Permissions: Fixed a permission issue in APIv2 where the system incorrectly checked for the "datasource" permission for Channel Family resources. Access is now correctly granted through the dedicated "channelfamily" permission.

25.12.01​

Bug Fixes​

• Fixed issue where sign in provider would be wrongly set to Google for new users. It now respects the sign in method used.

25.12.02​

Bug Fixes​

• Corrected cleanup logic to ensure Docker images remain, as long as they are being used by active workloads.

25.12.03​

Bug Fixes​

• Fixed Frontend bug: Fixed an issue regarding the google analytics API, that was causing the Frontend Build to fail

Restricted Administrator Access in the Energyworx Console​

Any user can now be configured as admin. When a user is first whitelisted, they are configured as non-admin. Admin access can only be granted individually, and not through User Groups in the Administrator -> Whitelist -> Details page. This can only be done by an admin, so you will first need to request admin access through the service desk.

Self-service client credentials setup in NUI user settings​

Client Credentials can now be created through the NUI settings page. This means it is no longer necessary to contact the service desk to get credentials to connect to the platform.

Notable Improvements & Fixes 🛠​

🚀 Platform & API Improvements​

• Smarter Datapoint Attributes (DPAs): Datapoint Attributes symbols, used in DPA enums, are case-insensitive. It is not possible to use the same symbol with different cases in the same DPA.

• Real-time Annotation Updates: Annotations are now updated in the table and chart after editing

• Improved documentation of APIv2: Removed invalid descriptions and added more descriptive titles for some API calls.

• Improved Audit Event Search: Fixed bug that prevented search by subjects (ie: flow id) in the audit events from the Datasource Details page

• Improved Transformation Configuration: Adding a transform rule to a static field in version and valid from in Channels and Tags is now supported.

• Task Creation Fix: Fixed an issue that prevented tasks from being created in flows when using the AbortOnError setting.

⚠️BREAKING CHANGE⚠️:

Tag Versioning Requirement: You cannot update only a Tag's valid\_from to a later date via the API or pipeline (ingestion/flow). To update the valid\_from to a later value, you must force a new version by modifying any other property in the same request. If necessary, you can then make a subsequent update to revert that property back to its original value.

🖥️ UI improvements​

• Chart Toggles Consistency: Enabling or disabling a channel or annotation in any chart in the original console is now always done through the square next to its name

• Consistent Menu Order: The menu layout in the new console now exactly matches the original console for better consistency.

• Knowledge Base: Knowledge Base page now opens in a new tab

• Faster Transition Between Original and New Console: Updated loading screen and much improved loading times for the new console.

• Dashboard Widget Saving: Fixed an issue when adding the first widget to a new dashboard wouldn't save the string replacement

💻 ewx-cli​

• Non-Admin Access: The CLI now skips admin-only resources when the current user is not an administrator, preventing errors.

• Who Last Updated What: The Upload Overview in the CLI now includes a helpful feature showing who last updated the resource.

• Namespace Alias Fix: We resolved an issue related to the namespace alias functionality in the CLI.

What's New ✨​

• Detailed Audit Event Filtering: You can now easily filter Audit Events in the New console by specific dates and times, giving you more granular control over your searches.

• Enhanced Datasource Tracking: Our BigQuery Datasources table now automatically tracks new datasources as they are ingested and processed. We've also added a table for simpler querying of the most up-to-date datasource versions.

• Improved Platform Link Sharing: All links to platform resources now consistently include namespaces. This means when you share a link with a colleague, they'll land directly on the correct resource without needing to manually switch namespaces.

• Flexible Trigger Timezones: When setting up triggers, you can now specify a timezone. This ensures your triggers run at the exact time you intend, based on your local timezone, rather than defaulting to UTC. (Existing triggers without a specified timezone will continue to use UTC).

Custom Market Adapters (API-Only)​

You can now customize Market Adapters directly on the platform to meet specific customer requirements. This allows for direct code modifications and additions.

Please Note: Currently, this feature is only available via APIv1. Support for custom Market Adapters will be added to the User Interface in a future release.

Enhanced Security:​

• File Uploads: Our API now actively blocks the upload of files with potentially malicious extensions, bolstering platform security.

• Data Disclosure: We've taken steps to further secure sensitive data by no longer disclosing Gunicorn in server headers.

Core Platform Updates:​

• GKE Version Update: The minimal required GKE version has been updated from 1.33, bringing general platform improvements and stability.

• Deployment Optimizations: We've made improvements to our deployment processes for greater efficiency.

Notable Improvements & Fixes 🛠​

Platform & Console​

• Case-Sensitive Datasource IDs for Tags: Fixed an issue where tags were not retrieved correctly if stored with an inconsistent datasource ID case.

• Dashboard Widget String Replacement: Fixed a bug that prevented all occurrences of a query string from being replaced in dashboard widgets.

• UI Flow Update Feedback: Corrected an issue in the UI where successful flow updates were shown even when a required rule parameter was missing.

• New Tab Functionality: Buttons in the console now support middle-click to open views in a new browser tab.

• Reduced UI Switch Time: Significantly reduced the time it takes to switch between the New and Old User Interfaces (OUI/NUI).

• Datasource View Time Range: Fixed an issue where the time range selection for timeseries in the datasource view was not correctly affecting the results.

• Channel Families List Fix: Resolved an issue that prevented the listing of Channel Families via the API endpoint.

• Accurate Collective Trigger Audit Events: Fixed an issue that prevented Audit Events from Collective Trigger pipelines from being written to BigQuery, especially when deadlines were set in the past.

• Accurate Tag Creation Feedback: Corrected an issue where "successful" messages were displayed even when an API error occurred during tag creation using reserved keywords.

Data Management & Rule Framework​

• Flexible Trigger Execution: Introduced a bypass_decision_tree and send_collective_trigger and CrunchTrigger objects. This allows for more dynamic control, enabling you to skip decision tree evaluation based on external logic.

• Data Ingestion Reliability: Resolved an unexpected error that could halt execution during data ingestion, ensuring that all data ingestion triggering flows now complete successfully.

• Improved Data Collection Retry: Added a retry mechanism for internal exceptions in collect and adapt functions of the Market Adapters (MA).

• Large Market Adapter Message Handling: Market Adapter messages exceeding the internal limits are now split and offloaded to Storage, where they will be processed normally by the pipeline.

• BigQuery Timeseries Fix: Resolved an issue where a cached classifier prevented timeseries data from being correctly written to BigQuery.

APIv2​

• Consistent APIv2 Timestamps: All timestamps returned by APIv2 are now consistently at microsecond precision, even if the most precise number is in seconds. This provides a uniform data format.

• APIv2 Datasource Naming: Datasource creation in APIv2 now only allows alphanumeric characters and underscores for storage reasons. Special characters are no longer permitted.

• APIv2 Empty Value Responses: Fixed an issue where APIv2 list endpoints were returning empty string for next page token instead of omitting empty values from the JSON response.

EDM packages:​

Dashboard Widget Accuracy: Corrected a widget to accurately count only METER datasources when displaying the number of datasources with critical events per day.

Pytest-bdd-ewx Improvements 🧪​

We've added several new steps to our pytest-bdd-ewx plugin, plugin, enhancing our testing capabilities for data editing and history verification:

• Datapoint Editing Scenarios: New steps to facilitate various datapoint editing tests: Edit single or multiple datapoint value, attribute or annotation values.
• Edit History Verification: Check for datapoint versions for continuous and scenario flow channels after editing. Confirm the previous value of a datapoint value, attribute or annotation in the edits history (with optional verification of edit reason and user email).

A full list of the available steps can be found here

What's new​

API v2 Enhancements:​

We have made several enhancements to our API v2 to give you more control and visibility:

• Start Flows with API v2: You can now start flows directly through our API v2.
• Flow Metadata Search: A new endpoint has been added for searching flow metadata, and it now returns flow configuration information.

Rule Framework:​

The rule framework has also been enhanced with new capabilities:

• Custom Logging from Rules: It is now possible to create audit events directly from your Transform Rules, by simply using the self.rule_logger in a transform rule.
• Split Step Monitoring: Rules in your flows can now retrieve the amount of messages that were generated during the split step in the ingestion process. Please refer to the documentation for details on this new feature in the market_adapter_split_count section.

Data Management:​

Here is a list of improvements related to data management:

• Advanced Search Feedback: Advanced search now provides more details if your query is not valid.
• Timeseries Support: The platform now ensures microsecond support for timeseries indices. Please note that flows and ingestions will continue to support millisecond precision at this time.
• Process Monitoring: We have added new "In Progress" metrics to Process Monitoring, allowing you to easily track the number of process instances that have started but not yet finished.
• Deprecated Client Removed: The deprecated ewx-client has been removed in this version.

What's Available in the New UI:​

The following functionalities are available in the new UI, which can be used by enabling beta features:

• Searching for datasources
• Managing channel classifiers (and underlying channel families)
• Creating market adapters
• Managing live flags and process monitoring
• Viewing audit events (Note: Links still lead to the classic UI for now)
• Using enerGPT, our in-house AI assistant

Integration tooling​

We've made major improvements to our integration tooling, with a special focus on making your development and testing workflows smoother.

Enhanced Pytest Plugin:​

Our pytest-bdd-ewx plugin is now more powerful, allowing you to easily test a wider range of API v2 endpoints. You can now:

• Validate Datasource Tags: Check and verify tags and their attributes on a datasource, just as you do for API v1.5.
• Create Tags: Create tags with specific valid_from and version fields directly from your tests.
• Verify Channel Attributes: Easily confirm the number of datapoints, attributes, and annotations for a channel in the Timeseries API v2 response, with support for filtering by date range, channel family, and more.
• Validate Search Results: Ensure that your datasource search returns only the specific datasources you are looking for.
• Monitor Live Flags: Get a live flag by name and use it to filter timeseries data, verifying the amount of flags per channel.

A full list of the available steps can be found here

ewx-cli API v2 Support:​

We've added initial support for API v2 in ewx-cli, starting with Channel Families. To use this new functionality, you'll need to update your ewx-cli credentials. We've added a new add-apiv2-credentials command to simplify this process for existing projects.

• Note: To use this new functionality, API v2 credentials must be created by an admin and added to your configuration.
• Backwards Compatibility: The ewx-cli won't fail if a Channel Family configuration is missing, but it will show a warning to help you identify missing configurations.

Notable Improvements & Fixes​

We've addressed several issues and introduced key fixes across the platform for a more stable and intuitive experience.

Platform & Console​

• Improved Search & Visibility: Advanced Search now provides more details on invalid queries, and the results table now displays all columns correctly.
• More Accurate Data: We've resolved an issue where older annotations would appear on newer datapoints. Also, timeseries data now has microsecond precision, and a bug was fixed where flow timestamps were incorrectly set to zero.
• User Experience: Linked the datasource flow in the TC to the flow configuration page, and fixed an issue where Datasource classifiers could not be changed more than once.
• Access Control: All users now have access to the EnerGPT page, and audit events are now properly filtered by severity.

API v2​

• More Accurate Filters: The Timeseries API now successfully filters on a specific version instead of just on ranges. We've also added support for comma-separated channel classifiers in API v1.5 timeseries calls.
• Improved Validations: Case-insensitive checks for duplicate names have been added, and we no longer allow symbols for 'non-enum' attributes, improving data integrity.

Data Management & Rule Framework​

• New Channel Family Naming: Channel Family names prefixed with EWX_ are no longer allowed to prevent conflicts between internal and customer-specific families.
• JSON Market Adapter: We fixed a bug in the JSON MA that prevented multidimensional JSON files from being ingested.
• Transform Rules: The get_existing_datasources method now returns an object for each datasource ID, including None for non-existing ones. Custom transform rules can now also be configured for a tag's valid_from field.
• Data Integrity: We've added validation to check for proper ISO datetime values for a tag's valid_from and version attributes.

ML Serving​

• More Cost-Effective ML Models: Our ML models are now more cost-effective when in an idle state, helping you optimize resource usage.

BREAKING CHANGE​

EQL queries: dropped support for aliasing of datasource_id table attribute in EQL. internal_datasource_id must now be used instead of datasource_id in EQL queries. This affects dashboard and advanced search queries.

25.02.01​

Bug Fixes​

• Updated search to allow for more "fuzziness"
• An issue was caused by an unexpected error in the code, which halted execution. All data ingestion that triggers crunch flows should now complete successfully.
• Added retry mechanism in collect and adapt MA for google TransportError exception, caused by NewConnectionError.
• Resolved an issue where the BigQuery ingest trigger schedule did not work due to a recent architectural change.

25.02.02​

Documentation

The Knowledge Base has been reintegrated directly into the platform and reorganized to provide clearer structure and easier navigation.

Bug Fixes​

• Fixed issue that prevented replacing all occurrences of a query string in a query.
• Added a bypass_decision_tree parameter to the send_collective_trigger method and CrunchTrigger object, enabling callers to skip the evaluation of any associated decision tree. This allows for conditional execution, where a decision tree may be defined but not executed based on external logic. The default behavior remains unchanged.
• Increased autoscaler resources

25.02.03​

Bug Fixes​

• Removed Billing account page from UI
• Fixed an issue that prevented "Created On" attribute from showing for Live Flags
• Added retry mechanism in collect and adapt MA for google TransportError exception, caused by NewConnectionError.
• Fixed an issue that was preventing Audit Events from Collective Trigger pipeline to be written to BigQuery.
• Fix an issue that was preventing to list Channel Families
• Fixed all/enabled/disabled buttons for flow design page
• Added retry error in run flow method for google Abort exceptions.
• Added missing permissions for Channel Families and Live Flag features. To keep viewing or managing these resource please ensure that the user groups have this permission added. These permissions are no longer managed separately from the User Groups in the Original UI. Note: Live Flags is categorised under the Computed Flags permission category.
• Fixed issue where Channels without DPAs could not be updated in the new UI

25.02.04​

Bug Fixes​

• Resolved an issue that prevented datasource view in the NUI from loading causing a permission denied error
• Resolved an issue where live flags using Enum DPAs would not evaluate properly when loading in a datasource
• Resolved an issue that prevented users from deleting Live Flag configs in the NUI

25.02.05​

Bug Fixes​

• Resolved an issue that prevented datasources with a dash in the ID from showing in the NUI

25.02.06​

Bug Fixes​

• ewx-cli documentation is corrected for deprecated commands
• Resolved multiple issues with permissions in the new API that did not respect the permissions of the user through their groups.

25.02.07​

Bug Fixes​

• Resolved issue that prevented ingestions from processing due to an issue with creating audit events from transform rules.
• The following issue was resolved: with the removal of the prepare pipeline in 25.02 (a part of the data processing pipelines), a bug was introduced that caused all messages to be routed to the processing pipelines of a single billing account. All messages that were processed still ended up in the correct tables, there were no issues in data separation on database level.

25.02.08​

Bug Fixes​

• Resolved issue that added an invalid suffix to some EQL table names that resulted in Table Not Found errors.

25.02.09​

Bug Fixes​

• Lists (in addition to dictionaries) are now able to be used in flow properties.

25.02.10​

Bug Fixes​

• Improved Transformation Configuration: Adding a transform rule to a static field in version and valid from in Channels and Tags is now supported.

• Improved Permissions Fixed issue when trying to retrieve timeseries for permissions with conditions.

• Improved Audit Event Search: Fixed bug that prevented search by subjects (ie: flow id) in the audit events from the Datasource Details page

25.02.11​

Bug Fixes​

• We now use a set of fallback machine types, so that if one runs out there's probably another. This reduces the risk of workloads not being scheduled to near zero.

25.02.12​

Bug Fixes​

• Improved Market adapter properties serialization: Fixed issue when trying to serialize Market adapter properties.

25.02.13​

Bug Fixes​

• Improved Timeseries Service: Fix issue when trying to set namespace in timeseries service.
• Improved XML File Creation: Fix issue when trying to create XML response files. This issue was caused by the mechanism to remove duplicated elements and attributes.

25.02.14​

Bug Fixes​

• Send bypass_decision_tree to CrunchTrigger model correctly.
• Fixed issue in Avro schema in the detectors writers.

25.02.15​

Bug Fixes​

• Improved BQ Streaming Service: Fixed a connection error in the BQ streaming process.

25.02.16​

Changes​

• Disabled EnerGPT: EnerGPT is no longer available. If you need to enable it, please reach out to the Service Desk to do it.

25.02.17​

Bug Fixes​

• Improved Timeseries Service: Fixed a bug where Datapoint Attributes of type 'timestamp' were being converted to naive timestamp objects, instead of remaining timezone-aware.

25.02.18​

Bug Fixes​

• Fixed Frontend bug: Fixed an issue regarding the google analytics API, that was causing the Frontend Build to fail

⚡ 10X Faster Search​

We're excited to announce the launch of our redesigned search experience, delivering dramatically improved performance and enhanced functionality.

You told us you wanted a faster search, our new search engine delivers results at least ten times as fast as the previous version in initial tests. In addition to more responsive filtering.

Key Features

• Interactive Tag Filtering : Click directly on tags to instantly add them to your search filter Multiple tags are automatically combined with "AND" logic
  • Tagged items are automatically copied to your clipboard for easy reference
• Familiar Classifier Filtering : Continue using datasource classifier filtering just as you did before
• Legacy Access Preserved : You can still access the original search page for "collections" and "bulk edit" functionality, by clicking "Go back to original view"
• Fuzzy Search: You don't need to worry about exact literal matching of the results, we'll show you the most relevant results first, but also potential matches. Additionally, It's now possible to search "ID AND tag.value=abc".

Getting Started

Visit the new search datasource page from the console sidebar to experience the performance boost immediately.

Simplified Market Adapter Configuration Experience​

New market adapter creation with intelligent pre-filling and guided configuration to reduce errors and technical complexity. Explore the new market adapter configuration view when creating adapters. Original view remains accessible if needed.

Key Improvement

• Pre-populated fields with validated options for the most common market adapters (CSV, JSON, XML)
• Reduced risk of configuration errors
• Minimised input requirements
• Reduced configuration complexity

Introducing Centralised Live Flag Management​

We're excited to announce the first full release of our Live Flags feature, expanding on the proof of concept previously delivered in 24.03. Live Flags now offers powerful cross-channel condition support, allowing you to create sophisticated validation rules that span multiple data streams.

Key Features​

Enhanced Cross-Channel Conditions

1. Multi-Channel Support : Create conditions that compare values across different channels
2. Centralised Flag Management : View and manage all configured flags from a single dashboard. Add new flags with the prominent "Create New Flag" button
3. Complex Condition Building : Combine conditions using AND/OR operators for precise validation rules
4. Flexible Flag Types : Configure datapoints as Valid, Invalid, or Custom with corresponding colour indicators

Powerful Visualisation

1. Colour-Coded Datapoints : Instantly identify valid and invalid readings in charts and tables
2. Toggle Visibility : Show or hide flags directly from the datasource view

Real-World Applications Examples​

Meter Reading Invalid Meter Read Conditions:

• Energy consumption (Channel A) shows activity BUT occupancy sensor (Channel B) indicates vacant property
• Meter reading shows consumption BUT circuit status (Channel C) indicates disconnection
• Reading exceeds historical maximum by 300% AND no change in customer profile
• Meter diagnostic channel shows error code concurrent with reading timestamp

Advanced Cross-Channel Use Case Examples

• Load Balancing Validation : “Invalid Reading” Flag on readings where phase balance exceeds threshold across multiple channels
• Revenue Protection : “Custom” Flag when consumption channel shows minimal use but auxiliary channels indicate activity
• Equipment Performance Monitoring : Flag when efficiency ratio between input and output channels falls below expected value

​

​

Getting Started​

To access the new Live Flags feature:

1. Navigate to the Smart Integration menu
2. Select “Live Flags”
3. Browse your existing flags or click "Create New Flag" to configure cross-channel validation rules

Known Issue

When assigning Datapoint Attributes of type enum, then the console at the moment expects the enum numerical value (and not the enum display text), we’re working on improving this user experience.

Documentation Updates​

• Users can now access our knowledge base from the platform’s side menu
• Renamed "Run" section to "Flow" and "Integration" to "Transformation Configuration" in API v1.5 docs
• The API documentation is now sorted alphabetically, and APIs descriptions are visible again

Added Partners Tools

• Transform rules can now import other rules (including any other type, as long as it derives from AbstractRule)
• Now pytest-bdd is available for the clients in GAR (python-shared repository) of the global project
• Added Cucumber-style BDD statements for pubsub client usage in testing pytest-bdd-ewx framework that can be used in automated testing
  • Native support for pubsub client usage in Behaviour-Driven Development (BDD) tests
  • Streamlined test scenario development for pubsub-based workflows
• New slice_by_version method for the TimeseriesData object. When calling this function with a version timestamp, it returns a dictionary with the version and a TimeseriesData object containing only the data for that specific version. If no version is supplied, the dictionary contains a TimeseresData object for every version in the data
• Introduced a new method to the TimeseriesService called get_versions that can retrieve multiple versions of timeseries data
• Added a new argument for the TimeseriesData.as_df function called multiple_versions. When this is set to True, it will return all available versions of the data as a multi-index DataFrame. When multiple_versions is set to False, it will only return the latest version of the data as a single-index DataFrame. It is set to False by default
• Added validations to create_task method in the rule framework. Now this function will validate:
  • Required fields are filled in:
    • title
    • task_type
    • status
    • priority
  • Converts single item types to lists for corresponding fields (aggregation_configurations, tags, taskboards)
  • Type validation for all the fields
• Introduced new way of stopping flows. This new method added the ability to persist the flow data that was saved up to the point where the flow is stopped by the user code
• Added in-memory caching implementation to apiv2
• Finalise the EnerGPT chatbot and release of the first version

API Changes​

New Endpoints​

• Added new API endpoint for datasource details that correctly splits Channel Properties from Tags

Modified Endpoints​

• Channel classifier IDs are now case-insensitive
• Fixed issue with index_time Lesser and LesserOrEqual filters not working correctly. This change enforces that the timestamp provided is in the RFC3339 format and in UTC
• Added API validation for channel families
• Consistently returns 404 when no channel families are present

⚠️ Deprecated Functionality​

We've removed deprecated API endpoints:

• GeoCodeBoundary
• Raw metrics
• Flow metrics
• TransportAdapter
  • You can use Energyworx Integration components

TL;DR: Technical Improvements​

Performance Enhancements​

• Added search by tag functionality to APIv2. Can reach up to a 10x speed improvement compared to the current API for certain cases Reduced the minimum Number of Pods used by the API Workload, resulting in leaner platform footprint
• Parallel flow execution & optimized resource usage across the platform
• Improved chart loading times and smarter API scaling

Security & Infrastructure:​

• Centralized policy management with Open Policy Agent integration
• GKE upgraded to v1.31+ with next-gen cloud infrastructure
• Modern machine types for all workloads, improving overall system efficiency

⚠️ Breaking Changes​

1. Removed Channel Family POST and PATCH via API v1. GET functionality remains to maintain support. Channel families are now created and maintained via API v2
2. Changed partition field from to insert_ts . This partition scheme gives more flexibility so it can be queried by seconds on the insertion time of a givenrow. Queries in EQL or Dashboard can use this new field. _PARTITIONTIME will be deprecated according to our deprecation policy, however, it can still be used until the moment it is deprecated

Notable Improvements and Fixes​

Data Management​

• Fixed issue with datapoint edits incorrectly using continuous mode when datasource flows are setup
• Fixed issue with edits not being loaded on timeseries data for datasource flows when no source channel data is present
• Original Datasource view - Fixes an issue where editing a datapoint with '0' value, removed the datapoint from the editing view
• Fixing restoring historical versions of flow configurations for datasource flows
• DPA of type "long" properly shows negative values now in the original console datasource view
• Fixed APIv2 issue where Annotations would not be returned
• Fix to set DPA LONG attributes to a string when rendering on FE instead of a number which avoids NaN errors
• It is possible to add data from other datasources to self.dataframe when using store_timeseries
• Fixed issue with writing NaN Timeseries values using API v2
• Editing data points value from the chart works as expected

User Experience​

• Channel classifiers page now show user friendly names for datapoint and unit types instead of technical names.
  • BEFORE: "CC_TEMP_A_" → AFTER: "Temperature Reading"
• Added a loader to channel family creation page for better user experience
• Adds tags in the expandable details of each datasource
• Under the user profile namespace selector, namespaces available to the user are sorted alphabetically
• Formats month(s) the same as version(s) for Channel Family retention policy
• Loading more results by "Show more" button no longer refresh the whole page on new console
• Improved console handling of invalid inputs for adding datapoint attributes to channel classifiers via new view, now a proper error message shows up
• Previously, widgets without a configured time period would incorrectly display a default timeframe in the information panel when a widget description was present. This could mislead users about the data's time scope
• Eliminated unnecessary Tags endpoint calls when changing dates on the datasource page
• Clarified what types of values are affected by edits by splitting the audit event to report the number of datapoints and the number of annotations edited instead of displaying them together
• When a flow tries to import a rule that doesn't exist an audit event with error message is shown
• Task descriptions rendering are no longer limited to 1k characters
• Corrected inconsistent highlighting of editable cells in the New User Interface (NUI)
• Invalid dates no longer accepted in the calendar on audit events page new view
• Fixed an error in the labelling of AND and OR in Channel Families
• Dashboard graph widget rendering as expected
• Created on date on file manager is showing correct date, Created on date am and pm is shown as expected
• A graph widget query startdate is based on UTC
• Fixes some datasource detail quirks, such as:
  • Load All Channels button should be disabled if all the channels are loaded
  • Date picker filter isn't using user settings preferences
  • Larger font on graph
  • Removes datasource view accordion in settings
  • Removes feedback X for source datasources not editable feedback panel

System Performance​

• Addressed a caching delay affecting the recognition of newly added Data Point Attributes (DPAs) in channel classifiers on API v1.5
  • Before: New DPAs were not immediately recognised, causing ingestion failures
  • After: Reduced latency in DPA recognition, enabling faster file ingestion
• Eliminated unnecessary Tags endpoint calls when changing dates on the datasource page
  • Before: Changing dates triggered redundant Tags endpoint calls
  • After: Reduced network calls, improving page load performance
• Changed the behaviour of internalising queries from Advanced Search
  • Audit events table queries now preserve original field names
  • Example: "id" remains unchanged in audit events queries
  • Multi-table queries continue existing internationalisation process
  • Reduced unnecessary query translations
  • More precise query handling for audit event searches
• Corrected unexpected timezone handling for timestamp-based Data Point Attributes (DPAs)
• Propagating the google exceptions (DeadlineExceeded, RetryError) to be retried
• Fixes on VDS execution:
  • API request is asynchronous for those long-running aggregations
  • Performance optimisations on aggregation execution
• Moved the user timezone to UTC for proper queries on the user dashboard
• Adding retry in prepare source data logic
• Fixed duplication of tasks statuses and task types when uploading their configs with ewx-cli
• Fixed issue when creating/updating rules. Now we check the status code of the response, and return the correct error message based on the response data

25.01 Minor Release 1​

Highlights​

• Improved Data Point Editing : Fixed issues with negative values and enhanced editing experience
• Enhanced Search Capabilities : Channel filters now work with both Display Names and Technical Names
• Better Tag Management : Improved visibility and functionality across the platform
• UI Enhancements : More intuitive colour selection and timezone consistency

Fixes & Improvements​

Data Management​

• Negative Values Support : Fixed issue where numbers couldn't be edited to be negative during datapoint edits in the NUI
• Channel Family API Compatibility : Added support for APIv1 created Channel Families in APIv2
• Tag Search Enhancement : Channel Properties are now returned by the tag search endpoint (datasource/search) in APIv1
• Datasource Classifier Management : Fixed issue where classifiers couldn't be changed twice on Datasources

User Experience​

• Display Name Filtering : Now it's possible to filter legend channels by their display names, not just Technical Names
• Chart Visibility Improvements :
  • Usability improvement to selectable colors in the chart legend
  • Eliminated invisible colors for better chart visualization
• Time Zone Consistency : Timeseries data time zone now matches dashboard graph's x-axis, which shows UTC time
• Dashboard Results Limit : Increased the configurable results limit in dashboard widgets to 1000

Tag & Property Management​

• UI Improvements for Property Keys :
  • Corrected display issues with technical property keys
  • Fixed tag width display problems
  • Added direct links from property values to related datasource detail pages

System Performance​

• Channel Visibility Caching : Default visible channels are now cached at the classifier level, ensuring accurate channel lists
• Timeseries Data Loading : Fixed issue with loading timeseries not appearing when edits were excluded
• Flow Labeling : Fixed labeled flows to use the correct workloads

Platform Cleanup​

• Legacy Component Removal : Removed transport functions from the platform navigation for a cleaner interface

Developer Notes​

• Fixed issue with loading timeseries for TIMESERIES_VIEW_BASIC and TIMESERIES_VIEW_INCLUDE_FLAGS filters
• Tags are now properly added via ingestion flow for datasources created via API v2
• Fixed internal issues with version handling when listing timeseries with annotations

25.01 Minor Release 2​

Highlights​

• Custom transform rules can now be configured for a tag’s valid_from field in the Transformation Configuration.
• Fixed an issue where datasource search on the console would not work without providing a datasource classifier filter, now it is possible to search without it.
• Users can view flags as disconnected points on the chart
• General availability of EnerGPT
• Added retry for pubsub exeception in market adapt logic
• Fixed an issue where querying timeseries didn't show all Datapoint Attributes.
• Fixes audit event api call loops that occurred when API returned 200 OK status but error response
• Fixed bug in JSON Market Adapter to allow the ingestion of multidimensional xml files.
• Idempotency for storing the tags once when the file is ingested multiple times.
• Cleaned up the chart export to include only the necessary data.

25.01 Minor Release 3​

Highlights​

• New Search, datasource and refreshed views available in preview

What's New​

Major Features​

1. EnerGPT Preview​

We're excited to introduce EnerGPT, our new AI-powered assistant designed to help you build and configure the Energyworx platform more efficiently. This preview release brings intelligent assistance to streamline your configuration workflow.

Key Features:

• Interactive platform knowledge assistance. Try asking: when do I choose a continuous flow over a scenario one?
• Configuration and rules guidance. Try asking: how can I store my Datapoint Attributes in a rule while removing empty entries
• Release notes exploration. Try asking: what were the highlights of 24.03 release in terms of timeseries management
• Natural language interaction with platform documentation

Benefits:

• Reduced configuration time
• Faster access to platform knowledge
• Immediate answers to platform-related questions

Preview Phase Details:

• Available in the new console view (beta features must be enabled)
• Token limit: 5,000 tokens per user during preview
• Focused on configuration assistance and platform knowledge

How to Get Started:

• Enable beta features in your user settings
• Access EnerGPT through the new console view
• Start asking questions about platform configuration, rules, or recent release notes

Note: As this is a preview feature, we encourage feedback to help shape its development. The current token limit is designed to help us evaluate usage patterns and ensure optimal performance. You can send your feedback directly via email to communication@energyworx.org, or create a change request ticket through our portal.

2. Channel Family Retention Management​

We're introducing comprehensive data lifecycle management through Channel Families, enabling automated enforcement of retention rules. This feature helps organizations optimize storage costs and maintain compliance requirements.

Key capabilities:

• Configurable Retention Policies

  • Version count-based retention
  • Time period-based retention
  • Combination rules (either limit can trigger retention)
  • Default 10-year retention policy for existing classifiers

• Automated Policy Enforcement

  • Google garbage collection integration
  • Up to 5-day completion window for cleanup
  • Automatic version management

• User Interface Integration

  • New console UI support
  • Channel Family management interface
  • Visual policy configuration

Implementation Notes:

• Channel Family assignments are permanent due to storage architecture
• Channel Family technical names should be no longer than 5 characters
• It's recommended to keep the number of channel families in your design less than 10 for most use cases. if you feel that you require more than 10 channel families, please reach out to us (either via support or directly to your Energyworx contact) and we are happy to work with you on the best practice implementation.
• Changes require Service Desk support
• All existing Channel Classifiers start with default retention

3. Business Process Monitoring Alerting - Get alerted when your business process has an issue​

We're adding alerting capabilities to the Process Monitoring feature. Now the users can be notified by email when a metric of their choice matches given criteria. Key capabilities:

• Support for the number of successful and the number of failed instances metrics (more to be added in future updates).
• Convenient way of defining criteria by selecting a metric, operator and providing a threshold value to be checked.
• Support for multiple timezones to match user workflow.
• Ability to specify multiple emails.

How to use

To set up a notification the user provides a condition to be checked (a threshold value and an operator).

Email can take up to 5 minutes to be sent out once the metric values are matching the condition. The user provides a list of email addresses where the notification supposed to be sent.

4. Task Creation in Rules​

We've introduced the ability to create tasks programmatically within rules, offering more flexibility in workflow automation:

Capabilities:

• Direct Task Creation
  • Use self.create_task within rules
  • No pre-configuration needed in flow design
  • Automatic subject relationship maintenance
• Comprehensive Property Control
  • Title and description
  • Task type and status
  • Priority levels
  • Assignee management
  • Task board assignments
  • Custom tags
  • Aggregation configurations
• Integration Features
  • Idempotent task creation
  • Automatic source tracking
  • Flow metadata linking

Example Usage:

self.create_task(title="Investigate High Usage", 
                 task_type="validation", 
                 status="new", 
                 priority=TaskPriorityEnum.HIGH, 
                 assignee_email="analyst@company.com", 
                 description="Usage exceeds threshold by 50%", 
                 taskboards=[123])

We suggest using this over configuring task in flow designs as users are limited to one per flow.

5. Enhanced Permission-Based Navigation​

The platform now features a dynamic navigation menu that adapts to user permissions:

• Dynamic Menu Rendering
  • Automatic permission-based visibility
  • Empty category hiding

Important Notes:

• The menu automatically updates if permissions change
• Administrative functions are no longer organization-restricted
• All menu items correspond to specific permission sets
• Empty categories are hidden to reduce interface clutter

6. Search Improvements​

We've upgraded the Search Datasources view with several user experience enhancements;

Long headers now gracefully truncate with tooltips, all timestamp displays are converted to local time with clear timezone indicators, and tag cell values can be copied with a single click. Additionally, we've added a quick-access button to the original datasource view while maintaining all existing search functionality.

Technical Improvements​

1. Performance Enhancements​

Infrastructure:

• Increased stability in Kubernetes scaling mechanisms
• Improved entity writer with transactional commits

API and Caching:

• Enhanced API caching mechanisms
• Query Optimisation
  • Improved data retrieval patterns
  • Enhanced filtering capabilities
  • Better handling of large datasets

2. Security Updates​

Authorisation:

• Open Policy Agent Implementation: We're working on revamping the authorisation model in APIv2 to allow for faster and more flexible security compliance and policy management, in this release we laid out the technical foundation on the API.

Audit and Compliance:

• We have updated audit event log when Uploading SOAP file, so filenames are included now

3. Configuration Management​

BigQuery ingest trigger:​

For power users using ewx-cli for repeatable deployments; String replacement is available for the BigQuery ingest trigger schedule. For this, a new property 'bq_template_mapping' was added. This property should be of format {'{KEY}': 'Value'} where it will replace any instance of '{KEY}' in the BigQuery query by 'Value'. This will be done automatically for {PROJECT_ID} and {NAMESPACE}.

Chain Rules​

Chain rules (bulk, conditional,chain) now have two new parameters, start and end date. If there are not given, the triggered flow will either use the dataframe indexes, if not empty, or the last two years. When deploying this fix for the first time to a new project, the rules configs need to be updated, using the api calls, by the support engineer doing the deployment:

{
"name": "start_datetime",
"displayName": "Start Datetime",
"widgetType": "DatePicker",
"valueType": "datetime",
"defaultValue": "",
"optional": true,
"description": "Start datetime for triggered flow"
},
{
"name": "end_datetime",
"displayName": "End Datetime",
"widgetType": "DatePicker",
"valueType": "datetime",
"defaultValue": "",
"optional": true,
"description": "End datetime for triggered flow"
}

Additionally, it's possible to use now comma-separated string for multiple datasource and flow configuration ids for bulk_chain_flow and conditional_chain_flow rules.

Advanced Search​

The flow_stats table is a new table that contains information on the duration of every rule in a flow. It is available in advanced search. Note that before this feature can be used, the namespace needs to be bootstrapped. More info on how to use this table can be found on our documentation page.

Deprecation Notices​

Tag Retrieval Function Deprecation​

• Deprecated: get_tag function
• Replaced by three new functions:
  • get_latest_tag_version
  • get_active_tag_versions
  • get_all_tag_versions
• Timeline:
  • Deprecation: 24.04

API Changes​

New Endpoints on APIv2​

Channel Families API

• Retention management endpoints
• Configuration endpoints

Policy Management

• Role management endpoints
• Permission configuration endpoints
• Policy enforcement endpoints

Modified Endpoints​

• Channel Classifiers
  • New field for assigning a channel family to a channel classifier

User Interface Updates​

New Features​

1. Search Enhancement

   • Introduction of a new datasource search view

   • Enhanced filtering capabilities

2. New Datasource View

   • ****Show/Hide All buttons are now visible by default in the datasource legends UI, removed from the Advanced section. Buttons maintain existing functionality and styling while being more accessible to users.
   • In response to partner feedback, we've expanded the chart zoom capabilities in the New UI to provide more precise and intuitive time range controls. These new options complement the existing zoom levels to offer greater flexibility in data visualisation. New Zoom Options:
     • Current Month: Shows data from the 1st of the current month up to the present moment
     • Last Month: Displays the complete previous calendar month
     • Last 7 Days: Provides a rolling weekly view

3. Audit Events View

   • Audit Events has a new checkbox that allows users to search through audit event properties through the UI. This could be used for example to search through rule logs. Note that this may have a performance impact on search results.
   • Trigger schedules will report errors in audit events when certain issues occur:
     • Technical name not existing
     • Wrong trigger schedule version
     • Incorrect values of parameters
     • Missing mandatory parameters

UX Improvements​

• Dashboards: Set the column width in result tables on the Dashboard to auto instead of fixed sizes.

• Dashboards: Fixes an issue where visual components in the dashboard quick settings would overlap under extreme browser zoom and resize conditions. The fix ensures:

  • Proper component spacing is maintained at all zoom levels
  • Time frame editor remains fully usable during window resizing
  • Quick settings panel maintains correct layout regardless of browser settings

• New Datasource View: Tag rows wrap longer strings instead of scrolling

• When a flow tries to import a rule that doesn't exist an audit event with error message is shown.

Notable Fixes​

Data Management​

1. Timeseries Management
   • Fixed an issue where deleting a timeslice group only removed its latest version. Now, deletion properly removes all versions of a timeslice group with the same name, preventing unexpected version restoration when reusing names.
   • We've improved how Datapoint Attributes (DPAs) are displayed in the datatable view, making it easier to access and understand attribute information while maintaining existing functionality. New Features:
     • Consolidated Attributes Column: New dedicated column showing all attributes for each channel
       • Grouped presentation of attributes per channel
       • Highlighted styling for better visibility
       • Can be hidden when not needed
       • Existing Views Maintained:
     • Existing Views Maintained:
       • Individual attribute columns remain available
       • Per-attribute filtering and sorting capabilities preserved
       • Original column-per-attribute view continues to function as before
     • Channel Pivot Improvements:
       • All attributes automatically group when pivoting on a channel
       • Annotations summary remains available during channel pivot
       • Maintains existing annotation functionality
       • Quick access to complete attribute set without additional clicks
   • Enhanced timezone support for the new datasource view
   • Fixed an issue where missing preferred timezone settings triggered a non-fatal error in the Original UI
   • Corrected unexpected conversion of DPA timestamp timezones. Please Note: Any attributes with timestamp values need to be converted to Datetime_utc in the transformation configuration rules.
   • Resolved a bug where accessing a non-existent property in user settings caused unexpected behaviour
   • Annotations can now contain Infinity as a value.
   • Empty values show up as false and values that cannot be set to false in datasource table
   • It is possible to use dynamic channels with different Datapoint Attributes that are described in the Transformation Configuration. However, if there is a missing Datapoint Attributes, it won't raise any error.
2. Channel Management
   • Better dynamic channel handling
   • Improved DPA support
   • Enhanced classifier management
3. Process Handling
   • Resolved an issue in handling existing file names when storing files from flows.
   • CSV files with one column are now detected correctly in Transform Configuration view.
   • We've enhanced XML file handling in Transformation Configurations by adding automatic column detection capability. This improvement streamlines the configuration process for XML data sources.
   • This addition makes working with XML files as straightforward as working with other file formats in Transformation Configurations, reducing setup time and potential configuration errors.
   • When in a Transform Configuration, the datasource classifier has an unknown value, an error message will be raised, specifying it is an error in the datasource classifier and not the channel classifier
   • Market Adapter Import - Enhanced Error Reporting Fixed an issue with Market Adapter imports by adding audit event errors when imports fail, providing clearer visibility into import failures and making troubleshooting more straightforward for both users and support teams. The change ensures that:
     • Failed Market Adapter imports now generate specific audit events
     • Error details are captured in the audit trail
     • Users receive clear feedback when imports don't succeed
     • Improved trigger scheduling
     • Better error messaging
4. Environment
   • Fixed an issue where bootstrapping new namespaces failed with a 404 error due to missing billing account association. The system now properly sets the namespace_id on the billing account during namespace bootstrapping step.

Looking Ahead​

Upcoming Changes​

1. View Deprecations
   • From 25.01, we're setting the following new views as defaults as they achieve feature parity for end users:
     • New Datasource Search View
     • New Audit Events View
     • New Channel Classifiers Management View
2. Future Enhancements
   • We're working hard on improving on our search performance in 25.01
   • APIv2 Timeseries, we're achieving x4 improvements for querying timeseries. For more, reach out to Energyworx Support Team

Migration Recommendations​

• Update to new tag retrieval functions in the Rule Framework

Known Issues​

UI Inconsistencies

1. New datasource view: Datapoint information under charts doesn't match clicked datapoint, so it may see incorrect data when analysing charts
2. Performance Variation in New Datasource View: While the new datasource view generally offers improved performance, some users may experience slower timeseries data loading times than expected. We are actively investigating to optimise performance across all use cases.
3. New datasource view: Switching between charts and table view resets edits in table view

Channel Classifier and Channel Family Management

1. Symbol Datapoint Attributes values displayed as "[object Object]" instead of actual value
2. UI lacks validation for invalid or empty input for Datapoint Attributes names
3. Channel family with negative or 0s retention policy can still be created
4. Reserved channel family name "ewx" shouldn't be allowed to be used, however it's now possible.
5. Cancelling the creation of a Channel Classifier leads users to incorrect page
6. API v2 doesn't check for duplicate channel family technical names
7. On APIv1.5: a. Unable to set retention policy for channel families in API v1.5 b. APIv2 GET Channel Family details endpoint doesn't support channel families created via API v1.5 c. We recommend using APIv2 for managing channel families

Minor Releases

Minor Release 1

• Fix the data loss that is caused by cloud-notification-router workload

Minor Release 2​

Bug Fixes

• Enabled/fixed SOAP endpoints in APIv1.5 and ensured SOAP functionality is routed via APIv1.5.
• Moves the chart datapoint click event to the rendered point instead of the chart object for details rendering consistency
• Changing deadline looper ctp limit to 1000 for E-redes envs.
• Fixed issue with datapoint edits incorrectly using continuous mode when datasource flows are setup.
• Fixed issue with edits not being loaded on timeseries data for datasource flows when no source channel data is present.
• Finds the proper datapoint clicked on the Datasource chart view
• Adds option to remove the empty value in a Dropdown component
• In channel family's creation page, the interface will only allow users to submit their request after all validation warnings have been resolved
• Fixed an issue where Datapoint Attributes with symbols type were visualised as 'object' instead of their proper value.
• Fixes an issue in channel family creation page where the display name was used for the technical name as well.
• Fixed collective trigger application packaging

Improvements

• Enhanced pagination for APIv2 timeseries queries
• Optimised resource allocation for datasource retrieval

Benefits

• Reduced number of API requests for multi-channel data
• More efficient data retrieval for large datasets
• Improved system resource utilisation

Minor release 3​

Bug Fixes

• Task descriptions rendering are no longer limited to 1k characters
• Adding retry in prepare source data logic
• Fixed issue when creating/updating rules. Now we check the status code of the response, and return the correct error message based on the response data.
• This includes a fix for dealing with the error messages when creating a new flow or tc, and also when updating a tc. We now check the status code of the BE request and based on it and the response data, we return the correct error message
• Added "No tags found" message when tag list contains only "group" tag.
• Added "ModuleNotFoundError" exception for csv files where the body starts with the letter "c" and contains more than one row.
• Namespaces now have an optional "sub_region" field that will override the "region" where a more precise location can be supplied for the BigQuery data. Values like "europe-west1" or "us-central1" are supported. For the full list of supported values, check this page: https://cloud.google.com/compute/docs/regions-zones#available. Please make sure to specify the region, not the zone ("europe-west1", not "europe-west1-b")

Improvements:

• Users can now access our knowledge base from inside the platform
• Formats month(s) the same as version(s) for Channel Family retention policy
• Improved API Pod Autoscaling based on the RPS (requests per second) coming to the API.
• Reduced the CPU and Memory resource usage of the API Workload.
• Reduced the minimum Number of Pods used by the API Workload.
• Updated the GKE version to be a minimum of 1.31.
• Added APIv1.5 scaling based on Google Cloud Load Balancer metric, to reduce the idle cost of the API workloads and to make the scaling faster.
  • We've implemented advanced scaling capabilities for API v1.5 using Google Cloud Load Balancer metrics, optimising both cost efficiency and performance during varying workload conditions.
  • Cost Change Impacts:
    • Significant reduction in idle workload costs
    • More efficient resource allocation during low-traffic periods
    • Optimised cost-to-performance ratio
  • Scalability
    • Faster scaling response during peak load periods
    • More responsive adjustment to sudden traffic changes
    • Improved handling of high-concurrency scenarios

Note: Changes may be less noticeable during light usage

Minor release 4​

Bug Fixes

• Fixed an issue with the writer component that affected how some tags were written into storage, that affected the flow metadata endpoint where retrieval of property values were missing in some cases

Minor release 5​

Bug Fixes

• Allow the rule parameter to be changed without the source code (until now, the rule parameter could only be changed if the source code was changed)
• Added more extensive API validation for channel families
• Consistently returns 404 when no channel families are present.
• Added API validation for channel families
• The form for adding tag on multiple datasources was changed because it didn't contain the property type and besides that, it was defined in controller. The type of tags was also changed, it should be sent as a list (it was defined as an object and I changed it to be a list).
• Created on date on file manager is showing correct date, Created on date am and pm is shown as expected
• Fixed an error in the labelling of AND and OR in Channel Families on the NUI.
• Removed the part that grants viewers access to energyworx users from create_new_project_in_google.sh
• Editing data points value from the chart works as expected.
• A graph widget query startdate is based on UTC

Minor release 6​

Bug Fixes

• Uncaught exceptions for particular combinations of uploaded files content's are dealt properly now and the file is properly processed.

Minor release 7​

Bug Fixes

• Original Datasource view - Fixes an issue where editing a datapoint with '0' value, removed the datapoint from the editing view
• Fixing restoring historical versions of flow configurations for datasource flows
• Enabled/fixed SOAP endpoints in APIv1.5 and ensured SOAP functionality is routed via APIv1.5.
• Dashboard graph widget rendering as expected.
• It is possible to search only a value and not a list

Minor release 8​

Bug Fixes

• New tag versions were created when the TC configuration was set to non-SCD. Now only one version remains.

Minor release 9​

Bug Fixes

• Flow results visualization missing backwards compatibility for some existing flow executions.
• Making sure get_existing_datasources returns an object for each datasource id (including None for non existing datasource ids).
• Correction of database instance location in processing pipeline.

Minor release 10​

Bug Fixes

• Fixed issue that prevented flow metadata being written to BigQuery.

Minor release 11​

Bug Fixes

• Added retry mechanism for google RefreshError Exception, in the collect Market adapter logic.

Minor release 12​

Bug Fixes

• Now it is possible to filter legend channels by their displaynames
• Channel Properties are now returned by the tag search endpoint (datasource/search) in APIv1.

Minor release 13​

Bug Fixes

• Now the time series data time zone matches the dashboard graph's x-axis, which shows UTC time.

Minor release 15​

Bug Fixes

• Custom transform rules can now be configured for a tag's valid_from field in the Transformation Configuration.
• Added retry for pubsub exeception in market adapt logic
• Fixed the remove and update tags for multiple datasources

Minor release 16​

Bug Fixes

• Custom transform rules can now be configured for a tag's valid_from field in the Transformation Configuration.
• Storing the tags once when the file is ingested multiple times.

Minor release 17​

Bug Fixes

• Longer task descriptions can be generated. Previous maximum size was 1500 chararcters.
• Creation of files from flows has gotten more robust.
• Add Service Account IAM to topic, that allows the Google Storage bucket to send a notification to timeseries aggregation topic.

Minor release 18​

Bug Fixes

• Added a retry mechanism to resolve data adaptation failures caused by network exceptions.
• Resolved an issue that prevented the bigquery_ingest trigger from functioning without a Market Adapter assigned.
• Updated AG Grid Key.

Breaking Change​

So if your eql query has no set insertion_date or _PARTITIONTIME parameter, and you’re querying on September 3rd, 2024, the results will only include data inserted after September 3rd, 2023. However, the data within those results can still refer to older dates (for example, if you’re backtracking to calculate billing for a past period).

If the date is specified, then the behaviour is the same as before.

The namespace needs to be bootstrapped for queries without a insertion_date or _PARTITIONTIME parameter to work.

Benefits:

1. Increased querying speed
2. Enhanced data safety
3. Expected decrease in unintentional BigQuery cost and usage

Change in default behavior:

EQL queries without a specified insertion_time or _PARTITIONTIME will now only include data inserted within one year from the query date by default.
Example of new behaviour:

If an existing eql statement is ran on September 3rd, 2024, without specifying insertion_time or _PARTITIONTIME, results will only include data inserted after September 3rd, 2023.
However, the data within these results can still refer to older dates (e.g., for backtracking or historical calculations).

Consistency with previous behaviour:

If insertion_time or _PARTITIONTIME is explicitly specified, the behavior remains the same as before.

Increase querying speed and add data safety, the platform now exposes insertion_time for all tables available to eql. An additional benefit is an expected decrease in unintentional BigQuery cost and usage

The change in behaviour is that now eql queries without a specified insertion_time or _PARTITIONTIME will only include data that has been inserted within one year from the date of the query by default.

2. Flow Duration Limit Warning​

We're introducing a new behaviour to help our partners build more efficient and robust flows. This update adds warnings for flows that exceed certain duration thresholds, promoting better performance and resource utilisation.

New Behaviour​

• 5-Minute Warning An audit event is generated when a flow's execution exceeds 5 minutes.
• 10-Minute Warning An audit event is generated when a flow's execution exceeds 10 minutes.
  In a future release, this will become an error event, and flows exceeding 10 minutes will be automatically aborted.

Benefits

• Early identification of performance issues
• Opportunity to optimise business rules
• Improved overall system efficiency

Technical Details

Duration Measurement:

• Ingestion and ingestion flows: Measured collectively
• Chained flows: Measured individually

Looking Ahead
These warnings are designed to help partners optimise their flows proactively. By addressing performance issues early, partners can ensure their processes achieve business goals efficiently and avoid future disruptions when stricter limits are implemented.

We encourage all partners to review and optimise any flows that trigger these new warning events.

3. New Units available in Channel Classifiers​

We've expanded the unit options available in channel classifiers:

• Gas Datasource Type: Cubic Meter per Hour (m3/hour)
• Water Datasource Type: Litre per Minute (L/min)

4. Knowledge Base Access​

• Users can now access the Energyworx Knowledge Base directly from within the platform.
• No additional login or authentication required.
• Simply click on the "Knowledge Base" item in the side menu to open it in a new tab.

5. Platform Version Visibility​

• The current platform version is now visible within the console.
• Users can easily reference the deployed version by looking at the bottom of the side menu.

6. New Alpha Datasource Search View​

• We're developing a refreshed search page with a focus on simplicity and reduced clutter.
• An alpha sneak peek of this feature is available to users with the 'beta features' flag enabled.
• This new view represents our ongoing efforts to enhance user experience and streamline data access.

7. New Datasource View Improvements​

Edit Reason Requirement​

To enhance security and compliance, providing an 'edit reason' is now mandatory when:

• Editing a datapoint value
• Modifying annotations
• Changing Datapoint Attributes

This change ensures a clear audit trail for all data modifications in the new datasource view.

Note: We've renamed 'value' to 'datapoint value' for improved clarity.

Annotations and Datapoint Attributes Enhancements​

• Both zero '0' and 'null' values are now inspectable from the details panel.
• Users can click on datapoints and their annotations from the chart when the annotation is activated.

Tags Improvements​

We've made several updates to improve the tags functionality:

1. Layout Optimisation:
   • Up to 4 tags are now displayed in one row, maximising screen usage for power users.
   • Tags are expanded by default for increased glanceability.
2. Simplified View:
   • The 'Versions' button is only visible when more than one version is available for a tag.
   • The 'Show Removed' button appears only when 'removed' tags are available.
3. Enhanced Styling:
   • Both tags' 'version' and 'valid from' are styled distinctly from other tag properties.
4. Future-Dated Tags:
   • Tags with a 'valid from' date set in the future are now hidden by default.

8. Process Monitoring Improvements​

We're excited to announce several enhancements to our Process Monitoring feature, aimed at improving clarity, usability, and error feedback:

Metrics Display Optimisation​

• The metrics results row has been redesigned for better space utilisation.
• Descriptions are now hidden behind tooltips, allowing more room for key metrics.

Time-Related Enhancements​

• Time zone information has been added to Start and End time metrics.
• For increased clarity, we've renamed the "Start Time" metric to "First process instance start time".

Real-Time Data Access​

• A new "refresh" button has been added to the metrics display.
• This feature ensures users can access the most up-to-date metrics at any time.

Improved Error Feedback​

• We've enhanced the system's response to non-existent flow configuration IDs in Transformation Configurations.
• When a user enters an invalid ID, the system now generates an audit event with a Warning level.
• This improvement provides clear and immediate notification of configuration issues.

Unshipped in this release

As part of our ongoing commitment to user-centric design and streamlined functionality, we've made several changes to remove or modify features that weren't meeting their intended purposes. This section outlines these changes and the reasoning behind them.

1. Transport Adapter​

• What : The transport adapter has been removed from the platform and isn't accessible from the console's menu.
• Why : It didn't provide the integration resilience we had aimed for.
• Alternative : We've developed a new component offering improved integration with external systems and hubs.
• Next Steps : For integration needs, please contact our service desk for an enhanced solution.

2. Advanced Search Panel Cleanup​

• What : Unused tables 'annotations' and 'annotations_bt' have been removed from the advanced search panel.
• Why : To simplify and streamline the search experience. We saw no active usage of these tables.

3. 'Generated ID' Helper Column​

• What : The 'Generated ID' helper column in the transformation configuration view is no longer available as an option for use with flow properties.
• Why : To reduce complexity and focus on more frequently used options.

4. User Settings - Channel Features​

• What : We've consolidated channel features into two main options: 'blacklisted channels' and 'Default visible channels'.
• Why : To simplify user choices and improve overall user experience.
• Removed : The "whitelisted channels" feature has been removed.

Notable Fixes

1. Task Management​

• Fixed an issue where task boards with no filter didn't load correctly.
• Task results limit is now respected in the task listing when set higher than 20 results per page.

2. Dashboard and Search​

• Resolved a layout issue in the Dashboard view affecting autoscaling of visual elements.
• Advanced Search now consistently supports using double quotes for queries, aligning with API usage.

3. Datasource Views​

Original Datasource View​

• Fixed an issue where editing the same datapoint multiple times without refreshing led to duplicated entries in the datatable.
• Resolved the problem of Datapoint Attributes with the same name being duplicated in all matching columns.
• Fixed the audit event stats display.
• Datasource view now shows the display name for a channel instead of the technical name when available.
• Addressed an issue where editing a datapoint with '0' value removed it from the editing view.
• Improved functionality to allow loading of Blacklisted Channels when needed.
• Fixed an issue where clicking 'switch to channel view' didn't behave as expected.
• Resolved an issue with edits and the pivot column causing user filters to reset.

New Datasource View​

• Fixed an issue where trying to open a datasource via the new console led to the channel classifiers page.

4. User Experience​

• Resolved an issue where users had to log out twice from the console (from both original and new views).
• Fixed an issue where updating the blacklisted channel list from the new user settings view didn't take effect.
• The new login view's popup prompting the user to login is now shown in the expected location.

5. Rule Framework​

• Datapoint values in dataframe can now be empty. The processing pipeline will continue to handle Datapoint attributes even if there are no Datapoint values.

6. Technical Improvements​

• Fixed an issue to prevent the simultaneous use of classifiers and channel_family in timeseries service. An exception is now thrown if both parameters are used. The filter by channel_family only has been fixed.

Minor Releases

Minor Release 2​

• When choosing default visible channels, only those channels are visible on the chart.

Minor Release 3​

• Fixed issue where old channel flow rules would fail to load in ingestions/flows.

Minor Release 4​

• Additional warnings on audit events when a flow is triggered from a transformation configuration, even when no flow configuration is assigned. This now results in a warning Audit Event to the user about the Flow Config missing.

Minor Release 5​

New Features and Improvements​

SOAP Integration Enhancements​

• Enabled SOAP response endpoint for improved integration capabilities
• Added channel properties and information completeness in responses
• Improved validation in store rules to verify DPA existence in channels

User Interface Improvements​

• Users with (billingaccount create, update, read) permission set can whitelist users for a namespace from a different billing account.
• We've re-enabled Channel Classifiers' new view after taking some time to work on issues that affected the user experience.
• When viewing a limited number of channels on chart on the datasource view, the user is able to see the channel name next to the line chart.
• We've reduced the number of visible markers when viewing a large number of datapoints on chart for better user experience
• Updated UI messaging for ad blocker extensions to use "whitelisting" terminology
• Added ability to delete assignee from task details
• Improved search functionality to prevent immediate result reset when modifying search terms
• Tags with future 'valid from' dates are now hidden by default

Data Processing and Performance​

• Fixed timezone handling for Datapoint Attributes timestamps. When storing Datapoint Attributes of type timestamp, the value needs to be converted to Datetime utc in transformation configuration rules
• Enhanced retries for failed timeseries loading operations in the Rules framework
• Improved file creation from flows robustness.
• Added cache expiration to channel classifier in timeseries endpoint on APIv2
• Improved handling of dynamic channels with different DPAs in Transformation Configuration

Chart and Visualisation Updates​

• Improved gap visualisation in new datasource view's chart.
• Enhanced new datasource view to show channel display names instead of technical names
• Timezones on the new datasource detail view uses user's timezone consistently
• Show/Hide All buttons are directly visible in datasource legends panel.

Bug Fixes​

• Fixed process monitoring metrics loading in process configuration
• Fixed an issue where bootstrapping new namespaces failed with a 404 error due to missing billing account association. The system now properly sets the namespace_id on the billing account during namespace bootstrapping step.
• Addressed logout synchronization between original and new consoles
• Fixed an issue where messages were processed with the default workload instead of the one corresponding to the label of the process

Technical Improvements​

• Enhanced API handling for flows without flow configurations
• Added Decision Tree ID to audit events for better traceability

Minor Release 6​

Platform Improvements​

Enhanced Audit Event Search​

We've expanded our audit event search capabilities. Users can now search through audit event properties and rule logs using a new checkbox option on the Audit Events page. While this feature provides deeper insights into your audit trail, please note that enabling this enhanced search functionality may impact performance.

Consistent Timezone Experience​

We've expanded timezone consistency across the platform. The system now respects user timezone settings in the new views for Channel Classifiers and datasource view, and the File Manager, ensuring a more coherent date display experience throughout the platform.

New Datasource View Enhancements​

Multiple improvements have been made to the handling of Datapoint Attributes (DPA). The system now correctly displays and enables editing of boolean "true" values, while also properly managing empty boolean values in the table view. We've also enhanced DPA value rendering in tables through improved channel key matching.

Misc.​

• Dashboard - The recipe listing functionality on Dashboards has been restored to full operation.
• Timeslice Groups - We've also resolved an issue where Timeslice Groups would enter a persistent loading state, preventing updates and deletions.
• Fixed a small issue affecting channel classifiers for French language users.
• We've standardised the handling of None and NaN values in annotation properties to prevent data inconsistencies.
• The platform now demonstrates improved resilience during high-volume ingestion scenarios, with enhanced retry mechanisms for cloud service interruptions.

Technical Improvements​

The platform's core infrastructure has been strengthened with improved error handling for cloud operations and enhanced data consistency across components. Various UI and visualisation issues have been addressed to provide a more stable user experience.

Minor Release 7​

Knowledge Base Integration

• Users can now access the Energyworx Knowledge Base directly from within the platform
• No additional login required
• Available through the side menu, opens in a new tab

Namespace Regional Options Enhancement

Namespaces now have an optional “sub_region“ property field that will override the “region“ where a more precise location can be supplied for the BigQuery data.

Values like “europe-west1“ or “us-central1“ are supported. For the full list of supported values, check this page: GCP region zones

Please make sure to specify the region, not the zone \(“europe-west1“, not “europe-west1-b“\)

Improvements​

Infrastructure & Performance

• Enhanced API Pod Autoscaling based on requests per second (RPS)
• Optimized CPU and Memory resource usage for API workload
• Reduced minimum number of API workload pods
• Updated GKE version to minimum 1.31

Task Management

• Removed 1k character limit on task descriptions
• Improved task description display capabilities

Bug Fixes​

Datasource View & Data Management

• Fixed datapoint edits incorrectly using continuous mode with datasource flows
• Resolved issue with loading datapoints edits on timeseries data for datasource flows
• Fixed visualisation of Datapoint Attributes with symbols type
• Corrected datapoint panel updates when changing clicked channels
• Improved chart datapoint click event handling for consistent details rendering

Flow & Rule Management

• Improved error handling when creating/updating rules with clearer messages
• Fixed issues with transformation configuration (TC) error messaging to be more user-friendly
• Fixed collective trigger application packaging
• Adjusted deadline looper CTP limit to 1000 for large customers' environments

User Interface

• Fixed an issue where Datapoint Attributes with symbols type were visualised as 'object' instead of their proper value.
• Fixed datapoint info accuracy under charts to match clicked points

EWXDEV-6648, EWXDEV-9281, EWXDEV-6638, EWXDEV-6648, EWXDEV-9281

At energyworx, we pride ourselves in providing immense flexibility for our users, and a need that comes often with a flexible platform is the need for transparency about underlying processes to empower our users to understand what’s taking place in the platform.

This release introduces a new console view for configuring and monitoring data processes within the platform. The Process Monitoring View provides users with a tabular interface to view details of processes and process groups, enabling real-time visibility into the status and progress of data flows.

What’s a Process?

A process is a series of data transformations in the platform to achieve business outcomes. This can range from simple ingestions to complex chains of flows until data is persisted.

What Processes are supported in this release?

In version 24.02, we support Market Adapters, Transformation Configurations, and Flows. Each tracked process resets daily. Continuous processes such as contract switching are tracked for the last 24 hours in this release. We’re working on adding more native support for other data pipelines like collective trigger and timeseries aggregation.

How can I track an existing process in the platform?

• Go to the process monitoring item on the sidebar
• Select the option to ‘track new process’
• Select a name that matches the business process you’re interested in implementing e.g. Daily Billing Process
• The name for a process may use alphanumeric, hyphens and underscores.
• Select the Market Adapters or Flow Designs that are part of the business process
• Under the hood, the process id will be passed to chained flows so metrics will include data from chained flows as well
• Select the metrics that fits your needs best, see documentation for full list of available metrics.
• After the processes ran -after process configuration is complete- will start being reflected in the Process Details Metrics
• Users can view process metrics from the Process details view.
• We’ve introduced a tabular view that updates with near-realtime metrics in the new frontend view.

Process Monitoring is Opt-in​

This feature is opt-in and uses the new console framework. To start using it:

• Pre-requisite : Ask your project consultant for a deployment of the new console views
• Pre-requisite : Enable beta features in the platform
• On the side menu of the console, navigate to Process Monitoring item menu

Better Compliance at a glance​

EWXDEV-9857

To ensure data security, we've added more compliance messages to audit events. These include:

Datapoint Edit

• Purpose : Track edits for transparency and accountability.
• Details : Includes flows ran (with custom parameter indication), user email, timestamp, and before-after states.
• For single datapoint edit, the event will include the datapoint value before and after edit, however
• For range or multi edits, the audit events will show details about the datapoints count and time range for the edits

Datasource Access

• Purpose : Monitor data access for security and compliance.
• Details : Includes datasource accessed, user email, and timestamp.

User Whitelist

• Purpose : Ensure proper tracking of user access changes.
• Details : Includes user email, timestamp, and namespace.

Group Change

• Purpose : Maintain accurate records of group membership changes.
• Details : Includes group involved, user email, and timestamp.

Permission Change

• Purpose : Audit changes to group permissions for security.
• Details : Includes group, changed permissions list, and timestamp.

Manual Ingestion

• Purpose : Track manual data ingestion for compliance.
• Details : Includes flow design, user email, and timestamp.

Testing Framework Package Availability​

EWXDEV-8657

For partners using the BDD testing framework to test their business process implementation, the package pytest-bdd-ewx is now available on the package hosting accessible to partners. Contact your project consultant for access details.

API Request Improvements​

EWXDEV-9242

API requests have been enhanced to make the console load faster. Authentication requests accompanying all API calls are cached on page, resulting in a smoother user experience in the datasource view.

Preview Feature - Computed Flags​

We've received feedback regarding the 'glance-ability' of datasource details. We're soft-launching our computed flags feature, which are live-calculated flags visualised without being stored.

The datasource view has been enhanced with a new data point status highlighting feature. This preview functionality allows users to quickly visualise whether a data point has a "good" or "bad" status, represented by green or red points respectively on the chart. Additionally, the status flags have been added as a new column in the table view, enabling easy filtering and identification of important data points.

The status can be configured in the channel classifier view using a flexible structure that supports multiple conditions with logical operators. Users can set conditions based on data point values, attributes, and a range of comparison operators (==, >=, <=, <, >, !=, ). Both "AND" and "OR" operations are supported, providing granular control over the status criteria.

Feature highlights:

• Quickly identify problematic data points with color-coded indicators (green for "good", red for "bad").
• Visual cues on chart and table update in real-time.
• Configure status conditions using a flexible structure supporting logical operators (AND, OR).
• Visual cues on charts make data interpretation straightforward, updated in real-time.
  Flag Conditions
• Tailored insights with channel-specific flag toggling using timeseries and datapoint attributes

This feature supports adding visualisation based on conditions evaluating timeseries on a single datapoint and its Datapoint Attributes value.

How can I test this feature?

• Pre-requisite: Enable new datasource view deployment and beta features.
• Configure the computed flags in APIv2’s channel classifier. Only API configuration is supported in this release
• Once the configuration conditions has been set, you can head to datasource details and see your computed flags in action

Note: this feature is experimental and computed flags can be hidden unless the condition is removed from the datasource view.

Under the hood​

Continuous Vertical Pod Autoscaler (VPA)​

EWXDEV-9662

This release introduces a Continuous Vertical Pod Autoscaler (VPA) that automatically monitors and adjusts pod resource allocation. The VPA checks for pod restarts caused by resource exhaustion and scales up CPU and memory resources to prevent future OutOfCPU or OutOfMemory conditions. By dynamically optimising resources, the VPA improves pod reliability, increases application availability, and reduces operational overhead.

Audit Event BigQuery Writer​

EWXDEV-10192: Improved the speed of writing audit events to BigQuery using new writers powered by Go.

Defect Fixes

Channels Visibility Preferences​

Blacklisted Channels Hidden by Default​

EWXDEV-8236, EWXDEV-9599

When a user sets a channel as a blacklisted channel in the user settings, the blacklisted channels are now hidden by default from the chart and its legend, as well as from the table in the datasource view.

Default Visible Channels​

EWXDEV-9250: When a user sets a channel as a default visible channel in the user settings, these channels are now shown by default in the chart and table in the datasource view.

Audit Event Details Update​

EWXDEV-9389: The file path subject is no longer shown in the audit event, and the flow_config_id is removed if it has no value assigned.

Streaming Ingest Service​

EWXDEV-10012: Fixed an issue where the streaming ingest service didn’t process messages correctly.

API v2 and New Datasource View​

Preventing Duplicate Entities

EWXDEV-9505: Creating a new entity in APIv2 will no longer overwrite an existing entity with the same name. The expected response is '409 Conflict' for new services like Process Monitoring, however existing services still return ‘500’ at the moment.

Pagination Issue Fixed

EWXDEV-9679: Fixed an issue on the new channel classifier view where pagination didn't load all data available in the platform.

Look n' Feel

• EWXDEV-9773: Added date filters for datapoints in the datasource view.
• EWXDEV-9844: Moved datasource properties to the top of the datasource view for better visibility.
• EWXDEV-9846: Cleaned up datasource view tabs, setting tags as the default tab and caching the API response.

Other Notable Updates

• EWXDEV-8548: Removed unused datasource table from the advanced search view under schema tab.
• EWXDEV-9923: Addressed GKE node scaling issue.
• EWXDEV-9926: Fixed contract payload processing issue in EDM.
• EWXDEV-10085: Resolved excessive GET requests issue when switching consoles.
• EWXDEV-10372: Fixed issue with excessive requests upon logout.
• EWXDEV-10477: Restricted editing of attributes to the appropriate channels in the frontend.
• EWXDEV-10478: Implemented advanced-search API call for EQL validation.
• EWXDEV-9754: Improved merge_datasources functionality in ewx-common.
• EWXDEV-10227: Added flag configuration to the Channel Classifiers via APIv2 for visualising "bad" values in the data.

Impact Analysis

Infrastructure Changes
- We’ve completed migrations of major API services from AppEngine to Kubernetes. This transition is seamless from a customer perspective as all routing has been adjusted accordingly.

- Audit Events Writer: Rewritten in Go for improved performance and faster visibility of new audit events.
- Continuous Vertical Pod Autoscaler (VPA): Introduced VPA to automatically monitor and adjust pod resource allocation, enhancing pod reliability, application availability, and reducing operational overhead.

API Breaking Changes
- APIv2 Datapoint Attributes: Whole float values are serialized without the additional .0 (e.g., 5 instead of 5.0).
- APIv2 and APIv1.5 Channel Classifier: Name is now case-insensitive; use display name for case sensitivity.

Compliance
- Audit Event Retention: Audit events will be retained for 1 year before being compressed and archived. But will always be available via Cloud Storage

24.02 Hotfix 1

Better Compliance Visibility:​

• Added audit events for various user actions, including:
  • Datasource access
  • User whitelisting/removal from a namespace
  • User addition/removal from a group
  • Group permission changes
  • Manual data ingestion
• Enhanced audit events for group changes to include a list of changed permissions instead of a full permission dump.
• Excluded detailed state changes from the "Datapoint Edit" action to avoid cluttering audit logs, relying on the edit history saved in Bigtable.
• EWXDEV-10993: [original console] Audit events are now logged after a datapoint edit with the option to run a flow is selected.

What’s Changed:​

• EWXDEV-8080:[original console] Fixed an inconsistent error message in the File Manager that appeared and asked for re-authorisation
• EWXDEV-9057:[original console] “Load channels” button will be set to disabled once all the channels are loaded and no more channels can be loaded
• EWXDEV-9491:[original console] Tag properties with type timestamp are now displayed with correct timezone in edit mode
• EWXDEV-10177:[original console] Fixed a visual issue where the legend filter showed "No results found" event when results are returned
• EWXDEV-10508:[original console] Editing a datapoint from continuous flow generates a new version as expected, see full impact analysis below for more details
• EWXDEV-10549: [original console] Users can now select blacklisted channels to load them even if hidden at first load
• EWXDEV-10627: [original console] Rule parameters of type timestamp default to the datasource timezone instead of New York timezone
• EWXDEV-10677: [original console] Task results limit is now respected in the task listing results when set higher than 20 results per page
• EWXDEV-11106: [original console] Using double quotes for queries in advanced search are now supported consistently with the API usage
• EWXDEV-10757: [original console] Removed “Readonly” filter on the file manager search.

Impact Analysis:​

Continuous Flow Editing Behaviour Change​

What's Changing:

Area Affected: Timeseries versioning and access within continuous flows. Secondary impact on external data access methods relying on flow metadata to access continuous flows.

Summary of Change: We're enhancing how manual data edits are handled in continuous flows. Edits will now create new data versions instead of overwriting the existing one, in-line with how continuous datapoint updates work in the rest of the platform.

Why It Matters:

Impact on Your Usage:

Performance: No impact

Scalability: No impact

Resource Utilisation: No impact

Cost Change: No impact

Technical Explanation:

This change aligns continuous flow editing behaviour with our documented behaviour. It ensures data consistency and allows for faster loading of datapoint in a continuous flow

Additional Notes:

Important Considerations:

Users accessing continuous flow data outside the platform via flow metadata will find this data inconsistent for all datapoints edits in a continuous flow. This access method is not supported.

The correct way to access continuous flow data is by requesting the latest available timestamp for the desired data point.

24.02 Hotfix 2​

We've realised a degradation in processing performance in 24.02 and this hotfix is tackling this issue first and foremost, here's a small analysis of what has changed:

We've changed our message deduplication using pod-name in the logic; we streamlined the logic into a fast access using locking mechanism, and removed additional inefficient logic.

Other Improvements

• Original Datasource view shows the display name for a channel instead of the technical name when provided by user.
• Dashboard Widget QuickSetting - Fields are resized for better user input experience
• Rule Framework - Datapoint values in dataframe can now be empty. Even if there are no Datapoint values, continue the processing pipeline will continue to process and handle the Datapoint attributes.

24.02 Hotfix 3​

Under the hood improvements mainly improving workloads that were stuck, and the robustness in writing data entities in atomic manner.

This hotfix also updates the Audit Event log message when uploading SOAP files, so that the filename is now shown as well.

24.02 Hotfix 4​

Rules Framework - Chain Rules Update​

Chain rules (including bulk, condition, and chain types) have been enhanced with new temporal control capabilities through the addition of two optional parameters: start_datetime and end_datetime.

New Parameters​

• Start Datetime
  • Controls the beginning of the time range for triggered flows
  • Optional parameter
  • Type: datetime
• End Datetime
  • Controls the end of the time range for triggered flows
  • Optional parameter
  • Type: datetime

Default Behaviour​

If datetime parameters are not specified, the system will:

1. First attempt to use existing dataframe indexes
2. If indexes are empty, default to the last two years

Technical Implementation​

Required API Configuration

Partners must update their rules configs with the following parameters:

    {
      "name": "start_datetime",
      "displayName": "Start Datetime",
      "widgetType": "DatePicker",
      "valueType": "datetime",
      "defaultValue": "",
      "optional": true,
      "description": "Start datetime for triggered flow"
    },
    {
      "name": "end_datetime",
      "displayName": "End Datetime",
      "widgetType": "DatePicker",
      "valueType": "datetime",
      "defaultValue": "",
      "optional": true,
      "description": "End datetime for triggered flow"
    }

Partner Actions Required​

No Action Needed

API Caching Fix​

Impact: Resolved caching issues affecting resource management for:

• Namespaces
• Users
• Billing accounts

This resulted in user rights inconsistencies for users working with multiple namespaces

Resolution: Fixed issues with resource fetching and saving operations, improving data consistency and reliability.

Other Changes​

• Fixed caching issue preventing namespaces from being correctly shown after being created initially.

24.02 Hotfix 5​

🔒 Important Infrastructure Update​

As part of our commitment to continuous improvement, this release includes critical infrastructure upgrades to our deployment management platform. Getting this update ensures your environment is prepared for upcoming feature releases and improvements.

UI Improvements​

• Fixed a display issue affecting whitelisted channel visualisation when using French language

Stability & Performance Improvements​

• Resolved an issue in handling existing file names when storing files from flows.
• Resolved an issue that affected resilience in the cloud-notification-router workload, the impact affected 1 in 5 million processes, and resulted in the data not being stored
• Improved retry mechanisms for temporary cloud service interruptions

Deployment Note​

The first time this fix is deployed to a customer environment it will recreate all pubsub subscriptions that support labelled flows. This means that all data contained in them will be lost. For this change to be deployed production environments must be completely shut down and the queues need to be empty before proceeding.

Console - New Datasource View 🎉​

We're excited to introduce our newly designed datasource view, engineered from scratch with a strong emphasis on usability, performance, and streamlined processes. This beta version introduces core functionalities including the ability to view, edit, and export timeseries, datapoint attributes, and annotations, as well as exposing datasource properties and tags view to the new API, ensuring a focus on essential components for a more efficient user experience. Additionally, the channels classifier view is now available for beta testing. Activate these beta features through your user settings on an opt-in basis; deployment and configuration is required on a project basis.

Key Updates Include:

• User Interaction Enhancements (EWXDEV-837) : Select and inspect datapoint attributes and annotations directly by clicking on a timeseries datapoint for an intuitive analysis experience.
• Multilingual Support (EWXDEV-5986) : Enjoy a consistent user experience across multiple languages, leveraging shared translations from the existing views and console.
• Improved Tag Visualization (EWXDEV-6840, EWXDEV-6842) : Experience an enhanced viewing of tags with increased real estate and a new tag component in our style guide. Users can view future tags as well as removed tags. Tags are available in the new datasource view
• Editable Timeseries (EWXDEV-8559) : Users can now edit timeseries, datapoint attributes, and annotations with a mandatory edit reason, currently available for continuous flows only.
• Clickable annotations and attributes in Datasource Details (EWXDEV-837) : The new datasource view allows users to inspect the Datapoint Attributes and Annotations by clicking on the timeseries datapoint, users can visualise Datapoint Attributes and annotations while timeseries is hidden

Please note, this beta view does not yet offer full feature parity with the existing view and is continuously being improved based on your feedback.

Customers may notice a refreshed appearance in data representation. This change is due to the implementation of dynamic scaling in our new console's graphs, which adjusts based on data availability to optimise viewing. This may result in different looking charts even when the data is identical

If you have any feedback for us on this view, please share it through this portal: Feedback Collection Portal

To go with the new Datasource View, we’re also introducing a new API. We’ve rewritten this API using the newest technologies and conventions resulting in a number of improvements including, importantly, speed. The new API is in beta with the corresponding new console view and is being continuously improved with additional features being added to match the offering of the ‘old’ API.​

Key Aspects of the API are currently:

• Datasource Functionality (EWXDEV-5857,EWXDEV-8198): Endpoints to query and perform writing/editing on timeseries data.
• Datasource and Channel Classifier Functionality (EWXDEV-5857,EWXDEV-8560): Endpoints to query and perform writing/editing on timeseries data. This also includes the new Datapoint Attributes functionality for usage with the Channel Classifiers.
• Tag Functionality (EWXDEV-8198,EWXDEV-6840): Endpoints to query and perform writing/editing on tag data related to a datasource.
• Timeseries Functionality (EWXDEV-5857,EWXDEV-8559): Endpoints to query and perform writing/editing on timeseries data. This also includes the new Datapoint Attributes functionality.

EWXDEV-7326 Console - Edit datapoint function displays who made a change​

Users are able to view edited datapoint with labels on the user and date of edit in Datasource view, both in the chart and table

EWXDEV-8340 Console - Limited support for Virtual Datasources interaction buttons​

In order to create a virtual datasource, it is necessary to create using the virtualdatasource/create endpoint and trigger it. This can be achieved either using the endpoint virtualdatasource/trigger or sending the triggerNow option during the creation.
Please Note: the trigger button (or virtualdatasource/trigger endpoint) is not working for the existing (before this release) Virtual Datasources at this moment.

EWXDEV-3006 Access Management - SAML Integration​

Single sign-in with SAML support. Users can log in in the energyworx platform by using their existing user account on other systems (support only for Microsoft and Google accounts).
This is supported through domain white-listing, meaning that all users belonging to a domain that is white-listed in the platform will be logged in automatically.
Automatic group-mapping is also provided. This enables the user to be assigned automatically upon log in to a group in the platform. This group can also be mapped to a specific group on the customer's own authorization system.

EWXDEV-6785 Flows - Support Chain flow(s) on Virtual Datasources result​

It is now possible to add a flow configuration ID for the vds_aggregation_ingest trigger schedule. This flow that will be executed on the resulting Virtual Datasources and will be triggered after a delay of one hour, to ensure that the data is in BigTable before the flow is executed.

EWXDEV-9334 New Channel Classifiers' Units​

It's now possible to create Channel Classifiers with datasource type “Signal” and new unit types: dB, dBm, Hz and Electric Current

EWXDEV-8207 Rules Framework - New load timeseries service​

Introduced the new 'TimeseriesService'; the replacement for 'AbstractRule.load_side_input' and 'AbstractRule.load_timeseries'.

⚠️ WARNING: With this change, the following methods in 'AbstractRule' are deprecated and will be removed in 24.04:

'load_side_input'

'load_timeseries'

'add_side_input_requirement'
Using any of these methods in 24.01 will create a deprecation warning audit event the first time they are used within each unique flow.
With 24.04, all rules that still use any of these methods will no longer function and need to be refactored!

EWXDEV-9097 Rules Framework - Deprecation of send_bulk_trigger​

The 'AbstractRule.send_bulk_trigger' method has been removed. Use the 'AbstractRule.send_collective_trigger' method instead.

EWXDEV-8981 Under the Hood - Move APIv1.5 to new k8s cluster​

Infrastructure change: The lionshare of the api endpoints will no longer be served from Google App Engine (GAE) but from a Kubernetes service running in GKE.

The API endpoint is now ah.<gcp_project>.ewxapis.com instead of api.<shortened_gcp_project>.energyworx.org

Energyworx will schedule a session with you to cover the following implications of this more cost-efficient setup:

Nonstandard networking on production environments could be affected - This needs a session with the integrator to verify any breaking changes.

As scaling has always been done for us in GAE, this will mean that for each project we will need to find the right setting with this new k8s implementation.

EWXDEV-9359 Under the Hood -Performance improvements on how the timeseries data is written in our GCP databases.​

EWXDEV-9376: CLI's 'config download' command ('Manager.to_dir') can now properly deal with merging configs into a directory with downloaded configs.

EWXDEV-9562: Added support for decompressing writer messages in cloud storage writer

EWXDEV-8717 Automated Testing Framework - Replace configuration/authentication management in pytest-bdd-ewx with resource manager from ewx-cli​

Replaced the authentication services in pytest-bdd-ewx with the one from ewx-cli for APIv1:

This means that authentication locally can now be done without using environment variables and with the automated authentication that ewx-cli users are used to.

Any missing environment variables are now simply requested from the user to be provided from the command-line.

Namespace aliases can be provided as target namespaces as well.

Due to using the ewx-cli automated authentication, automated tests ran are now also marked as having been ran by the actual user instead of a default testing user, improving traceability and reducing security issues.

The configuration manager of ewx-cli can now be used within pytest-bdd-ewx

Configurations directories created with 'ewx-cli config download' can now be treated as proper configurations within pytest-bdd-ewx.

Several new step definitions were added where an automated test can initialize a resource manager from a configurations' directory. Users can both create a default resource manager available to all automated tests or a named resource manager only available to the automated test that initialized it. A step definition was also introduced that allows an automated test to upload all configurations available in a resource manager.

Many new step definitions were added for all supported resource types where the user can request from either the default or a named resource manager to upload a specific resource type with a specific name. This will upload that resource configurations plus any dependencies it may have, similar to the effect you normally get with 'ewx-cli config upload'. Step definitions also exist where a regex pattern can be provided that names of that resource type must satisfy, or when the user wants to upload all configurations of that resource type.

$-substitution is still fully supported. If a value in a configuration in a configurations directory has the substring {$test_timestamp}, then that will be replaced by the actual testing timestamp.

The old fashion way of writing automated tests is still supported, but it is recommended to start using the new step definitions with the ewx-cli's resource manager, as it is far more efficient and reduces the complexity of automated test suites.

New Fixes​

• When a new data is ingested for a datasource, the (previous) annotations are now correctly overwritten.
• Ingesting timeseries data that is not a float will now properly raise an error. Example: "Channel(s) TEST failed to transform for datasource(s): XXX, with message: One or more values in channel data cannot be converted to float"
• sourceClassifier attribute needs to be a valid channel classifier in order to run a flow, if not it will result in an error and the flow will not be executed.
  If sourceClassifier is not given it will use the source channel of the flow configuration id
• Fixed a small issue to ensure consistency between the default selected namespace on the new datasource view
• Fixed an issue on dashboards where chart widgets were empty on first load, or the legend section was shown as empty
• Dashboard table columns resizing to fit the data
• Fixed a bug where file tags where disappearing after ingestion
• Fixed an issue where Datapoint Attributes retrieved wasn't returned always in the same order from the APIv2
• Added audit event to specify which rule failed during prepare context. This way the user has a better understanding of which point in the flow there was an error.
• Fixed an issue where a flow failed if you assigned a task board in the task configuration.
• Fixed Redis IP being set incorrectly.
• Configured task board assigned to tasks created from flows
• In some cases, a Bigtable DeadlineExceeded error would end up in an audit event instead of being retried.
• Fixed an issue where users couldn't select a subset of the flows after editing datapoints
• Fix for errors querying FlowMetaData for specific datasources via API which do not have the readOnly flag set.
• Fixed the create/patch Channel Family endpoints
• Fixed loading of transformation configurations in API v1.5
• Make sure that timeseries retrieval is end-inclusive.
• Fix for flows not starting.
• Fixed an issue that prevented some specific filter combination for searching in File Management and Task Management.
• Fix task schemas in API v1.5
• Fix to use the default appengine service account for the python3 API service.
• Fixed issue with flows not loading.
• Fix flow start from task page
• Fix task filtering using statuses for API v1.5
• Task Boards are fixed and can now be updated.
• Always drop the version index label for the channel source data.
• Fixes issue with loading transformation configurations in the console.
• Fixed time formating to %Y-%m-%dT%H:%M:%S.%f matching what's used in production.
• Pass the data to the correct parameters and correctly format it
• Aligned the names for the task creation attributes.
• Split channel classifier and timeseries GET requests on the datasource detail page to prevent query parameter overloads.
• Fixed loading of tasks on the task flows endpoint.
• Make sure that nested rules can also be loaded directly.
• Fixed proper parsing of annotation fields in the streaming ingest
• Annotations are now added with their full name to self.dataframe
• Fixed issue with APIv1.5 for the file content endpoint.
• Fixed issue in creating legacy Virtual Datasources
• Table height adjustments ensure consistent display, regardless of row quantity.
• Datapoint Attributes should now no longer be visible in the frontend as unreadable byte values.
• Fixed issue with annotations not showing in the frontend console timeseries chart.
• Channels tab doesn't trigger datasource flows now
• Fixed an issue where users couldn't access Triggers that they were authorised for it
• Fixed an issue where conditionally chaining datasource flows still required a classifier
• Added standard utils back
• Fixed an issue where the defer-task-queue got paused after app engine was deployed.
• Resolved: Conditional flow rule inputs now fully editable, including setting Start Delay in chain flows.
• Fix issue of the store rule not storing Datapoint Attributes if the value of the Datapoint was set to NaN or None.
• Datastore timeouts in Streaming-Ingest no longer result in silent errors and will instead be retried by Pub/Sub.
• Added hotfix for the usage of dictionaries in annotations.
• Fixed an issue where the annotation used for creating tasks from a flow would affect the startDate subject of the task.
• Fix annotation editing via the console.
• Fixed a CORS issue that affected the smooth transition between the new datasource view and the existing view
• Solve the issue of potentially invalid cached model id
• Annotation and datapoint editing has been fixed to work for the same timestamp.
• Fixed Tags with Timestamp properties showing the property in the UTC timezone instead of the Datasource timezone.
• Fixed an issue where Timeseries could not be retrieved if a specific Service Account was not whitelisted to the namespace.
• Fixed API key usage for API v1.5
• File details now has a functional download link when clicking the file name.
• Front-end console adjustment so text isn't shown twice for datasource.
• Fixed a login issue that affected users that had not been registered in the platform yet.
• Fix for a bug where task search didn't work if there is no one assigned.
• Fixed bug preventing annotation/datapoint edits from being reflected in the chart prior to the edit going through.
• Fixed timestamp inconsistency in datasource properties where an edited datapoint's timestamp would always be displayed in UTC even when the user has selected another timezone (such as Europe/Amsterdam).
• Ensure that Tag objects are always created with lowercase names.
• duplication of the Virtual Datasources filter property fixed from frontend side with a limitation that it's not possible currently to edit it from UI. The root cause still should be fixed on the backend (as part of another ticket)
• Fixed datapoint editing timezones
• Modified bulk_chain_flow to accept datasource flows without a target channel classifier
• Fixed an issue where switching between visualisation types didn't apply to the chart
• Fixed issue that prevented the IAM Group create and update endpoints from showing up in API docs.
• Task descriptions originated from detector elements in flows will be limited to less than 1500 characters to prevent excessive storage use. Detectors information is already available in the flow metadata if needed.
• Fixed issue where searching for files was very slow for retrieving a large amount of files.
• Timeseries values that have been written with datapoint attributes can now be viewed in the console using APIv1.5 (without the datapoint attributes themselves). For retrieval of datapoint attributes, please use APIv2
• Added functionality for Collective Trigger to trigger single message after deadline has reached. This means that when this option is chosen every message that arrives after the deadline will trigger a flow with only the flow properties of this single message. The function send_collective_trigger has been modified to support this.
• Fixed a bug where Datapoint Attributes were not stored correctly in BigTable when the dataframe did not include the channel classifier, but did include the Datapoint Attributes.
• Fixed a bug where datapoint attribute columns were treated as channels, causing the serialization of the data to fail.
• Added workaround that removes duplicate timestamps that have the same version as well.
• All timeseries functions that retrieve existing data correctly processes Datapoint Attributes values
• Extend signedURL on API file manager model to work beyond 1500 bytes.
• Make sure that any channels created during 'apply' for a Datasource are also added directly to the Datasource to allow channel flows to be chained to them.
• Use pickle's protocol-2 for any trigger send out from streaming-ingest to allow Python 2.7 services to pick up such triggers appropriately.
• heartbeat channels are ignored for timeseries reading in APIv2
• Fixed issue where the billing account cannot be saved if no email domains are set.
• Fixed bug with triggers starting task archiving.
• Fixed dashboard charts to have proper legends with channel names.
• Added Datapoint Attributes to the datasource console page and added Datapoint Attributes editing functionality.
• Datapoint attributes can now be stored with values like float.NaN, np.nan, pd.NA and None. Those values will be filtered and will not be saved.
• Make sure to only store the latest version of each datapoint that has been marked for storage in the flow.

Impact Analysis​

Trigger Aggregation Deprecation Notice​

From release 24.01, we will remove the send_bulk_trigger function from the rule framework, for message orchestrating, we recommend using the send_collective_trigger function instead.

Package Deprecation Notice​

Starting February 2024, the existing ewx client python package that used APIv1 to connect to Energyworx API will stop receiving stability updates and will eventually be phased out, in favour of supporting the generation of new API clients based on our Partner's preferred language.

Please contact Energyworx support for assistance when deploying regarding these actions

Shutdown of GAE py3 API

We’re moving APIv1.5 to Kubernetes. This will be deployed automatically. However, two manual things need to happen:

• Old load balancer needs to be updated to route traffic to the Kubernetes service.
• Remove the GAE py3 service after deployment. So that no traffic can accidentally use the old service.

Cleanup App Engine execute_triggers
The execute_triggers has been moved to a terraform managed "Cloud Scheduler Job". It used to be a Google AppEngine's “Cron Job”. The old job gets now orphaned after deletion of the GAE service. It needs need to be cleaned up manually.

Re-index Datasources for Search functionality.
This release removes the default Kubernetes cluster used in the previous release and recreates it (ewx-unicorn). This thus deletes the ElasticSearch cluster and any data in it. This needs to be re-indexed.

Changelog​

24.01 hotfix 1​

• Big improvement to the datasource view speed. Mainly when timeseries and flow metadata is being retrieved.
• New Datasource View: Fixed an issue where source channels could be edited.
• New Datasource View: Added removed and created by fields to tags.
• Datasource View: Fixes an issue when searching legends by channel name.
• Increased the number of data queried on a datasource which allows large datasources to be loaded robustly
• Fixed an issue where timeseries wasn't visible in the platform when it contained empty timestamp values
• Fixed an issue of missing references for building configuration outside the platform

24.01 hotfix 2​

• Fixed an issue where blacklisted channels were not hidden in the new datasource view
• Added better exception handling for retryable errors in the decision tree execution.
• Flow properties for collective trigger now accept any type that is serialisable in python (with pickle)
• Validation logs in audit events now shows serialised timeseries data for each Channel Classifier in case of failure.
• Fixed a security issue with group permissions where blacklist conditions using datasources and tags weren't enforced
• Fixed an issue where the link to the energworx Service Desk in the Console was not working
• Fixed a pagination issue across refreshes on the new datasource view
• Providing an edit reason is now mandatory from new datasource view when editing a datapoint
• Long names are now truncated in the top bar breadcrumbs
• Unit types and intervals should be part of the channel classifier in the new Datasource View
• Fixed an issue in the new Datasource View where the tags version timelines wasn't responsive to the screen size
• In the new Datasource View, fixed an issue where showing removed tags filtered non-removed tags
• In the new Datasource View, channels are loaded automatically when “Get Data” is selected.
• In the new Datasource View, a datasource timezone is respected in the chart, table and tags properties
• In the new Datasource View, it's not allowed to edit source channels timeseries data
• In the new views, the namespace label is shown under the username in the top right of the screen

24.01 hotfix 3​

• Fixed an issue where Channel name from Channel Classifier was not provided, now it's working when Channel name was empty or None value by showing the Classifier's technical name.
• Fixed an issue where navigating between the current and experimental console view caused excessive numbers of requests.
• Channel Family can now be chosen using the API.
• Fixed an issue on the new datasource view where a new token was being requested for every API call.
• Fixed an issue where blacklisted channels were still shown in the datasource view.
• Corrected the assignment of roles and permissions in the identity service.
• Under the hood - Removed AppEngine Cron Jobs that caused trigger schedules to run in duplicate.
• Fixed an issue with channel classifier caching in the new datasource view wasn't utilised.
• Under the hood - Unified serialisation package use in the platform code.

24.01 hotfix 4​

• Fixed an issue in the new datasource view where the API's token refresh was called more time than necessary
• Fixed an issue when updating channel classifiers affected the channel classifier count
• Corrected issue such that APIv2 always loads the latest datasource version.
• Fixed issue where edits with the word _nan_ in them were not displayed via the api.

24.01 hotfix 5​

• Fixed issue where group permission properties could not be altered and fetch via apiv1.5.
• Added support for api_key_allowed for retrieving datasources via datasource/get/{id} route
• Datapoint Attributes interpret 0 value correctly as a number instead of None value
• Fixed an issue where load_side_input method in Rules Framework still use cache even when bypass_cache flag was set to True
• Fixed an issue where datapoint edits didn't behave consistently in APIv1.5 across different data structures when deserialising
• Fixed an issue where datapoint edits didn't show correctly in APIv1.5 and APIv2
• Under the hood: more robust writers quota

24.01 hotfix 6​

• Changed the parameter page size to load full versions of Tags in apiv2.
• In certain scenarios where Market Adapters require context data (e.g. ancillary datasources like a Meter for a Metering Point) the context data was too large, reaching the limits of the queueing system in between the pipelines of the platform. To solve this we applied compression on the exchanged messages in the queuing system.
• Fixed an issue where edits with zero-values did not persist in the table in the new datasource view.

24.01 hotfix 7​

• Fixed an issue with editing Datapoint Attributes via frontend where cells belonging to any channel could be edited for a row belonging to a different channel.

24.01 hotfix 8​

• Fixed an issue with ‘Abort on Error’ feature in Flows. If the error threshold is reached, a task will be created and flow execution will stop. Please note: All the data that was saved until that point will be persisted.
• It is now possible to add data from other datasources to self.dataframe when using store_timeseries.
• Fixed an issue when editing Datapoint Attributes and annotations on the same datapoint on the console in datasource view.
• Under the hood: It's possible to support push subscription for external components outside the platform
• Fixed an issue where the validation of the Datapoint Attributes data type didn't raise an error. It now causes the flow to stop executing with a suitable error message in the audit events.
• Added better logging in audit events when the market adapter fails to parse the data provided into energyworx domain objects.
• Cloud log error message when logging DeadlineExceeded error is more descriptive
• Datasource view in the console shows dates in Datapoint Attributes values parsed correctly
• New API Documentation URL is updated correctly to match the standard convention
• We've simplified the file management search by cleaning up the read-only/writeable filters.
• Under the hood: Channel classifiers are better cached in the processing pipeline for faster storage of Datapoint Attributes.

24.01 hotfix 9​

Infrastructure & Performance​

• Under the hood - Updated deployment software for improved reliability
• Fixed an issue where AH pods were entering restart loops due to probe settings
• Entity writer now uses transactional commits for improved reliability

Data Processing & Rules Framework​

• Messages in processing pipelines now include message IDs to prevent duplicate triggering
• File creation from flows is more robust, addressing issues with dead-letter queues
• Fixed precision of version_ts field in BigQuery tables for edited datapoints to maintain millisecond accuracy
• Rule Framework - Enhanced handling of empty datapoint values in dataframes
  • Processing pipeline now continues to handle Datapoint attributes even when Datapoint values are empty
• Chain rules (bulk, condition, chain) enhancement:
  • Added new start_datetime and end_datetime parameters
  • If not specified, the triggered flow will use either:
    • Dataframe indexes (if available)
    • Last two years (as default fallback)
• Data processing improvement: Rule parameters of type timestamp now default to datasource timezone instead of New York timezone

User Interface & Experience​

• Dashboard Widget QuickSetting - Improved field sizing for better user input experience
• Advanced search functionality now consistently supports double quotes in queries across API usage
• Simplified file management search by removing read-only/writeable filters
• UI improvements:
  • Fixed "No results found" message display when filtering channels in legend
  • Tag properties with timestamp type now display with correct timezone in edit mode
  • Load channels button is properly disabled when all channels are loaded

Logging & Monitoring​

• Added more detailed audit event logging for SOAP file uploads to show filename

24.01 hotfix 10​

Stability & Performance Improvement​

We resolved an issue that affected resilience in the cloud-notification-router workload, the impact affected 1 in 5 million processes, and resulted in the data not being stored

Schema Optimisation: Data Point Attributes​

As Energyworx platform aims for cheaper and efficient implementation, the schema optimisation is an initiative that focuses on how timeseries is stored in our platform, this results in two big changes: Data Point Attributes & Channel Families.

Data Point Attributes

Users who have a lot of attributes that belong to timeseries datapoint can now make use of Datapoint attributes; the concept allows for storing data of the following types [bool, enum, float, timestamp] up to 80% more efficient storage. Datapoint Attributes' timeseries can be read and written from the same places annotations are.

This feature isn’t compatible with existing channel classifiers without a migration, but new channel classifiers support it out of the box. Currently, from the API

Channel Families

Users are able to set up timeseries data that will be access frequently together under the same channel family, for optimised data loading experience.

This feature isn’t compatible with existing channel classifiers without a migration, but new channel classifiers support it out of the box. Currently, from the API

Timeseries Aggregation​

AKA: Virtual Datasources on BigQuery

Continuing with the new vds_aggregation_ingest trigger schedule that we introduced in the previous release, we improved it in this release such that it does not need a Market Adapter to be configured any more. This means that the aggregation of the timeseries is automatically creating the datasource and channels of the datasource from the results of the aggregation.

Collective Trigger Pipeline​

Our replacement for bulk- and aggregation- dataflows feature

We introduced a new service called The Collective Trigger. This service collects messages from flows (triggering flows) in the platform and groups them by an identifier called the Correlation ID. This new service will replace the existing bulk trigger.

When a certain condition is reached (e.g. a number of messages collected per Correlation ID), then the service will handle the start of a flow (a result flow) processing with the arguments passed on the Collective Trigger messages.

These messages can be sent by a rule being executed in a flow by calling the function send_collective_trigger. This function can set two main types of parameters:

• Flow execution parameters: the parameters for the flow execution, e.g. the flow design id, the flow properties, etc.
• Collective Trigger handling: conditions on how to handle the messages for a certain Correlation ID, e.g.: the amount of expected messages, the deadline expected for the desired amount of messages, etc.

The flow properties of the messages received are combined per Correlation ID and shared with the result flow.

There are certain conditions that can be set for handling the Collective Trigger. They can set waiting windows for messages of the same Correlation ID in two different ways:

The deadline: a timeout window which starts after the first message is received and indicates that a specific amount of messages for the same Correlation ID is to be expected before this time is reached.

The timeout in between messages: a timeout window that starts with each message and indicates how long it is expected for a new message of the same Correlation ID to arrive.

When a condition of the above is expired and a message corresponding to that same Correlation ID is received, then we define these messages as Late Data. We can configure several possible behaviors to handle this Late Data:

• Discard: simply discard the message.
• Trigger flow combining all data again: trigger the message again with the combination of all flow properties.
• Restart the window: restart the waiting window.

The following functions in the rule framework are affected by the implementation of this new feature/service:

• New send_collective_trigger function is created, this can be used from any rule to call the Collective Trigger function.
• send_bulk_trigger function is redirected to send_collective_trigger function.
• Following send_bulk_trigger parameters have been set to defaults and will be deprecated:
• number_of_current_items: set to 1
• trigger_late_flow: set to True
• send_crunch_aggregation_trigger is removed and any existing functionality has been migrated to send_collective_trigger.
• send_bulk_trigger function is marked as deprecated and will be removed in the future release.

Deployment Note: during the deployment of this release all business processes that uses the bulk functionality should be stopped. The business process can be restarted after the deployment is finished.

APIv1.5 - Migration to Python3​

Python2 is no longer supported by Google starting the 1st of February 2024, therefore an upgrade to Python3 is needed. Below is the stacks of technologies we upgraded to or now use in this migration:

• Python3.11
• Flask-openapi3
• Pydantic
• Google cloud ndb (instead of Google appengine ndb)

In this migration, we created new API blueprints which includes specific endpoints definitions. All request/response models were recreated using Pydantic and the database models in the controller layer were decoupled by introducing a separate pydantic models for request and responses. Last, we mapped these models with the ndb models. to_model() method is used to map a pydantic model to an ndb model before sending it to the service; and from_model() method is used to map an ndb model returned by the service to a pydantic model, which will be the response of the API.

This new API blueprints can be found in api_py3/packages/energyworx-console/gae/ewx_gae/api/v1_5

Edited Datapoints Audit​

In this release, we added the possibility to verify the previous version of an edited datapoint, along with important information about who did the edit, when and what value they edited a datapoint to.

This information is visible on the data table and chart, additionally, when a user clicks on a datapoint, edit information will be visible in the datapoint panel in Datasource View.

Fixes in this release 🪲​

• Fix an issue in timeseries aggregation where not all channels were created

• EWXDEV-8549: Fixes an issue in timeseries aggregation where the Name and ID in the aggregation configuration weren’t always persisted

• EWXDEV-7481: Fixes an issue in timeseries aggregation where channels with whitespaces in the column name weren’t created

• EWXDEV-7663: Fixes an issue where similar tasks weren’t grouped by in the platform

• EWXDEV-8225: Fixes an issue where changing the source channel for a channel flow before running it did not take effect

• EWXDEV-8589: Fixes an issue where file tags where disappearing after manual ingestion

• EWXDEV-8718: Fixes an issue where timeseries aggregation’s filter, the filter now supports using the same tag and property multiple times, enabling queries like tag.prop = A OR tag.prop = B

• EWXDEV-8719: Fixes an edge case where uploading files with no default tags caused an issue with ingestion

• EWXDEV-8734: Fixes an issue affecting the 'Run Flow' menu item from the flow metadata component.

• EWXDEV-8741: Fixes an issue affecting the creation of tasks from a flow.

• EWXDEV-8743: Fixes an issue in transformation configuration affecting dynamic channel classifiers with no display name

• EWXDEV-8756: Fixes an issue where in some cases, a Bigtable DeadlineExceeded error would end up in an audit event instead of being retried.

• EWXDEV-8757: Fixes an issue where timeseries aggregationtrigger didn’t create a Virtual Datasource when no chained flow was set

• EWXDEV-8782: Fixes an issue where Virtual Datasources filters weren’t displayed in the console

• EWXDEV-8780: Adds a daily timeout reset for collective trigger

• EWXDEV-8786: Fixes an issue where timeseries aggregation chained flows weren’t triggering underlying datasources on first trigger.

• EWXDEV-8811: Fixes an issue where running a scenario flow on edited datapoint breaks the datapoint edit API

• EWXDEV-8817: Fixes an issue with rendering API Documentation page description

• EWXDEV-8818: Fixes an issue affecting writing Detectors from flows that aren’t marked as readonly

• EWXDEV-8826: Fixes an issue affecting “channel family” create endpoint

• EWXDEV-8829: Fixes an issue where reloading a transformation configuration causes the page to be empty

• EWXDEV-8853: Deleted datasource classifiers aren’t displayed by default on Datasource Classifier page on the console

• EWXDEV-8854: Fixed an issue where flows with custom parameters failed to trigger

• EWXDEV-8862: Fixes an issue with flows not loading on Datasource view

• EWXDEV-8865: Fixes an issue affecting users running a flow from the task board

• EWXDEV-8866: Fixes an issue affecting users trying to add a lane to existing task board

• EWXDEV-8866: Fixes an issue affecting the status filter for task boards

• EWXDEV-8875: Addressed a workaround while loading datasources by using channel view as default for datasources with more than 12 flows.

• EWXDEV-8911: Resolved an issue preventing joining of tz-naive with tz-aware DatetimeIndex.

• EWXDEV-8913: Fixed an issue where Virtual Datasource created via API had no data.

• EWXDEV-8915: Resolved issues with loading transformation configurations with multiple channels in the console.

• EWXDEV-8916: Fixed issues related to loading Timeseries in the console.

• EWXDEV-8933: Standardised time formatting in Datasource Search to align with production standards.

• EWXDEV-8935: Fixed issues related to range editing multiplying for datapoints.

• EWXDEV-8936: Aligned the names for task creation attributes to resolve an error in GET/run/search.

• EWXDEV-8937: Split channel classifier and timeseries GET requests to prevent query parameter overloads.

• EWXDEV-8942: Addressed the issue of the missing priority field when creating tasks from a board page.

• EWXDEV-8961: Corrected an issue in the initialisation process for creating legacy VDS.

• EWXDEV-8962: Addressed the disappearance of the Load Channels button in the console when no flow timeseries is available.

• EWXDEV-8969: Ensured Datapoint Attributes are correctly formatted and readable in the frontend.

• EWXDEV-8970: Resolved a problem preventing the manual editing of boolean tag properties.

• EWXDEV-8972: Fixed an issue causing annotations to not display in the frontend console when no timeseries is visible.

• EWXDEV-8977: Modified the Channels tab to prevent triggering of datasource flows.

• EWXDEV-8989: Resolved an issue preventing admin users from accessing trigger details.

• EWXDEV-8995: Fixed an issue where conditionally chained datasource flows required a source channel.

• EWXDEV-8998: Corrected a problem causing the defer-task-queue to pause automatically.

• EWXDEV-9008: Addressed a UI issue affecting multiple flow selections in the console.

• EWXDEV-9009: Resolved a UI issue with dependent flow checkboxes in the console.

• EWXDEV-9014: Implemented a fix to enable unlinking tasks from a task board.

• EWXDEV-9031: Fixed a store rule issue not storing DPAs with NaN or None values.

• EWXDEV-9046: Addressed inconsistency in default channel view setting in the console.

• EWXDEV-9056: Ensured persistent display of the channel/flow view switch in the console.

Platform Stability Enhancements

We've addressed several critical issues to improve platform reliability and performance:

• Fixed DNS resolution failures for datastore operations
• Resolved DLQ message handling for stream-crunch operations
• Improved deadline handling for StoreData operations
• Enhanced flow view editing capabilities

Configuration Updates

• Updated E-REDES Monthly and Billing Scaling Parameters in infrastructure configuration

Fixes in this release 🪲

• EWXDEV-11444: Resolved message handling in DLQ for ewx-edp-dso stream-crunch
• EWXDEV-11639: Fixed flow view editing errors
• EWXDEV-11980: Addressed datastore.googleapis.com DNS resolution failures
• EWXDEV-12253: Improved deadline handling for data storage operations
• EWXDEV-12154: Updated scaling parameters for E-REDES billing system

Deployment Note: Standard deployment procedures apply. No service interruption expected.

23.03 Hotfix 45​

Bug Fixes​

[EWXDEV-12205] CTP Limit Adjustment​

• Changed deadline looper CTP limit to 1000 for certain environments, addressing CPES response status issues.

[EWXDEV-12253] Data Storage Timeout​

• Added retry in prepare source data logic to handle "Deadline of 60.0s exceeded StoreData failed" errors.

[EWXDEV-12432] Data Retention​

• Fixed data loss caused by cloud-notification-router workload, preventing datasource loss during high-volume ingestion.

Infrastructure Improvements​

[EWXDEV-12172] Cache Optimization​

• Increased Redis instance size to 5GB for all instances (more cost-effective than 4GB configuration).

23.03 Hotfix 46​

Bug Fixes​

[EWXDEV-12253] Data Storage Reliability​

• Added retry mechanism in prepare source data logic to address "Deadline of 60.0s exceeded StoreData failed" issues.

​

23.03 Hotfix 47​

Bug Fixes​

[EWXDEV-11230] SOAP Endpoint Integration​

• Enabled/fixed SOAP endpoints in APIv1.5 and ensured SOAP functionality is routed via APIv1.5, resolving ingestion issues with customer-specific workflows.

[EWXDEV-12449] Market Adapter Processing​

• Added retry mechanism for Google RefreshError Exception in the collect Market adapter logic, fixing issues with unprocessed customer's messages.

[EWXDEV-13276] Timeout Handling​

• Added ServiceUnavailable retry in prepare_source_data.py to address 60.0s timeout issues.

Improvements​

[EWXDEV-8113] Flow Management Enhancement​

• Introduced new way of stopping flows with the ability to persist flow data saved up to the point where the flow is stopped by user code.

23.03 Hotfix 48​

• Certain partners have access to get_datasource function is available in decision trees now.
• When all the requested data is present in the cache, return only the requested time window.
• Channel Properties are now returned by the tag search endpoint \(datasource/search\) in APIv1.
• More errors are retryable in the processing pipeline.

23.03 Hotfix 49​

• The VEE processes resources have been increased for large-use-case environments.
• Added retry for pubsub exeception in market adapter's logic

23.03 Hotfix 50​

• Adjusted the soap outbound request timeout times and persisted changes to the soap outbound task queue.
• Updated custom market adapters implementations for custom use cases

23.03 Hotfix 51​

• The Datasource Billing writers resources have been increased for large-use-case environments.
• Updated custom market adapters implementations for custom use cases

23.03 Hotfix 52​

SOAP endpoint creation now routes to the correct API backend service again. Creation of files from flows has gotten more robust. Added retry mechanism in collect and adapt MA for google TransportError exception, caused by NewConnectionError.

23.03 Hotfix 53​

Increased pod-custom-metrics-stackdriver-adapter resources

23.03 Hotfix 54​

Updated retry mechanism for google TransportError exception, caused by NewConnectionError.

23.03 Hotfix 55​

Allow Market Adapter Split messages to be sent to Storage when the message is greater than the Pubsub limit size. These messages will be processed by prepare pipeline normally.

23.03 Hotfix 56​

Fix Pubsub size message verification to allow the overhead of base64 encoding.

⚠️ Warning

Deprecation Notice​

We are in the process of deprecating ewx-client and would recommend using our public API for data access.

Platform Changes: Defects, Improvements, and New Features

Defects​

• When a new data is ingested for a datasource, the previous annotations are now correctly overwritten.
• Fix and provide a better error message when the user come back after being idle/away from the console for a while.
• Fixed bug where load-side-input would not always use cached data from ingestion or previous flow
• Blacklisted in user settings channels do not show anymore on the DS details page in "Channels" section
• After returning all the search result hits on Search Datasources page, if the user clicks 'View More', the results page does not reset anymore.
• Tag names and tag property keys are now checked for their values. Incorrect values will no longer be stored, but raise an error and Audit Event instead.
• Invalid datasource and channel classifiers in flow and transformation configurations now create Audit Events when attempted to be used.
• Invalid transformation configurations configured in market adapters now create Audit Events when attempted to be used.

Stories and Improvements​

• We streamline the user experience when the user tries to start running a flow, depending on which type of flow the user selects. If they choose to start a new datasource flow, then they would not see the source channel, but if they choose to start a new channel flow, then they can see (and select) and source channel. This is done to improve the user experience consistencies across platform.
• We streamline the user experience for the Flow Design when it is opened on Datasource Flow Scope, the source & destination channels are not visible anymore. This is done to improve the user experience consistencies across platform.
• Json format file is recognised and can be mapped in Transformation Configuration.
• All generic market adapters now check the extension of the file before ingesting
• Caching can be disabled for flow and decision tree resource configurations on staging environments.
• Reduce model inference time when iterated in a rule by implementing caching to fetch model by name.
• Caching can be disabled globally for all resource configurations on staging environments.
• Any non-source datasource now has a flow metadata property in the datasource view to show what the source was of that flow.

New Features​

We introduce new ML model training functionalities where the user can train their ML models using the data from the Energyworx platform. A query to prepare the data and a training script according to our training framework is required.

Currently supported ML frameworks are: scikit-learn, xgboost, and xgboost_multi_output. The xgboost_multi_output is required if the prediction produces more than one output at a time. We also support non-distributed hyperparameter tuning with hyper-opt. We also provide docker images / dockerfiles to test locally the user's code.

When the training succeeds, the trained model and its metadata is uploaded in the file manager. When the training job fails, the traceback info is uploaded as well. The ML framework also allows a flexible workflow where the user can train the same model with different data by configuring \sqlqueryfile\\ as filename in trainingflow parameters.

Argo Workflow​

The workflow yaml file to integrate different steps of training pipeline and manage internal artifacts and parameters between steps.

Includes an image with necessary permissions and packages and retrieval go script as an entry point taking sqlquery as input argument from workflow.

Allows to create a placeholder file in the job submitter. This is needed to allow successful runs of the step which expects an output. The output is needed in case of failure to pass the metadata to the exit hook.

Argo expects the artifact output, once configured, to always have the output artifact present even in successful runs.

Metadata Persistence​

Adds the metadata writer WorkflowTemplate. It receives a metadata file and copies it to a destination GCS bucket. This template can be deployed to the argo training installation.

Once the WorkflowTemplate is installed, its templates can be used in Workflows. You can use this workflow template on its own to test it without the other components.

BigQuery Usage Dashboard​

This dashboard can be used for detecting queries that use up a lot of slottime/bytes

Every Monday a report will be sent to customer about the previous week

New method to store timeseries data​

We introduced new methods for AbstractRule:

• store_timeseries(timeseries, channel_id=None, *, datasource_id=None, store_in_flow_data=True)
• store_annotations(annotations, channel_id, annotation_name=None, *, datasource_id=None, store_in_flow_data=True)

On top of that, we also introduced changes in the functionalities such as:

• Storing timeseries or annotations can be done independently of each other.
• Both methods can take a Series or a DataFrame.
• Timeseries and annotations can have any timezone now.
• Channels are only created on the datasource if actual data is required to be stored on them.
• A DataFrame of timeseries will store all columns independently on the channels which are the column names. Any channel classifier that does not exist on the namespace will raise an error and abort the flow.
• A DataFrame of annotations will store each column independently as annotations for the specific channel_id channel.
• Annotations can now have any name you want (series name or column names are used). Only requirement is that this name is unique.
• Annotations can be created on non-existing datapoints. This also means that gap annotations will now actually be stored on the gap itself instead of one annotation after the gap.
• Multiple annotations can be created on the same datapoint within the same rule.
• Multiple versions of annotations can be stored on the same timeseries version.
• Timeseries and annotations can be stored on ANY prepared datasource, bringing it in line with how you can add tags to any prepared datasource.
• Timeseries and annotations stored with store_XXX can optionally also be added to the flow data (which self.dataframe is created from), which is turned on by default. This only works for data stored on this datasource. Data stored on other (prepared) datasources will never be added to the flow data.
• Annotations stored with store_annotations are recognized by self.data_filter.
• Heartbeat channels are no longer supported.

VDS Improvements​

• Improvement VDS query validation check. First our console will try to validate the query input. In case the query is invalid but somehow managed to bypass the console validation, then the second (backend) validation will be triggered to notify the user in audit events that their query is actually invalid.
• Allow VDS to copy flow properties from input flow to output flow
• Allow user to modify VDS filter via API. The new filter only affects new flows run and not the existing crunched data.
• Allow user to set time limit for which a VDS is updated
• Allow user to add delay parameter to VDS when using chained flows for underlying datasource. The maximum time delay is 20 minutes.
• Allow user to set timezone for VDS. However, it is still possible for the user to view the VDS in timezones that are the default of the underlying data sources.
• Deprecate obsolete VDS API endpoints
• Add field for timezone for the VDS in Advance Search

Channel Classifier Display Name​

• Introduce a new parameter called display name for Channel Classifier
• This display name will be used in Channel Classifier view, Flow Design view, TC view, and Datasource view

Adding New Properties to Channel Properties​

• Add new properties valid_from, version and deleted to channel properties
• Add the new properties above to TC channels section
• Extend generic runbook to cover ingestion scenario to add channel properties

Datasource Classifiers Limit​

• Increase the allowed limit of datasource classifiers to 1000

Timeslice Group Versions Comparison​

• Improve the previous TSG versions' comparison UI.

SAML Integration​

Single sign-in with SAML support. Users can log in to the energyworx platform by using their existing user account on other systems (support only for Microsoft and Google accounts). This is supported through domain white-listing, meaning that all users belonging to a domain that is white-listed in the platform will be logged in automatically. Automatic group-mapping is also provided. This enables the user to be assigned automatically upon log in to a group in the platform. This group can also be mapped to a specific group on the customer's own authorisation system.