Skip to main content

Release Notes: Version 24.02

What's New

Process Monitoring View​

EWXDEV-6648, EWXDEV-9281, EWXDEV-6638, EWXDEV-6648, EWXDEV-9281

At energyworx, we pride ourselves in providing immense flexibility for our users, and a need that comes often with a flexible platform is the need for transparency about underlying processes to empower our users to understand what’s taking place in the platform.

This release introduces a new console view for configuring and monitoring data processes within the platform. The Process Monitoring View provides users with a tabular interface to view details of processes and process groups, enabling real-time visibility into the status and progress of data flows.

What’s a Process?

A process is a series of data transformations in the platform to achieve business outcomes. This can range from simple ingestions to complex chains of flows until data is persisted.

What Processes are supported in this release?

In version 24.02, we support Market Adapters, Transformation Configurations, and Flows. Each tracked process resets daily. Continuous processes such as contract switching are tracked for the last 24 hours in this release. We’re working on adding more native support for other data pipelines like collective trigger and timeseries aggregation.

How can I track an existing process in the platform?

  • Go to the process monitoring item on the sidebar
  • Select the option to ‘track new process’
  • Select a name that matches the business process you’re interested in implementing e.g. Daily Billing Process
  • The name for a process may use alphanumeric, hyphens and underscores.
  • Select the Market Adapters or Flow Designs that are part of the business process
  • Under the hood, the process id will be passed to chained flows so metrics will include data from chained flows as well
  • Select the metrics that fits your needs best, see documentation for full list of available metrics.
  • After the processes ran -after process configuration is complete- will start being reflected in the Process Details Metrics
  • Users can view process metrics from the Process details view.
  • We’ve introduced a tabular view that updates with near-realtime metrics in the new frontend view.

Process Monitoring is Opt-in​

This feature is opt-in and uses the new console framework. To start using it:

  • Pre-requisite : Ask your project consultant for a deployment of the new console views
  • Pre-requisite : Enable beta features in the platform
  • On the side menu of the console, navigate to Process Monitoring item menu

Better Compliance at a glance​

EWXDEV-9857

To ensure data security, we've added more compliance messages to audit events. These include:

Datapoint Edit

  • Purpose : Track edits for transparency and accountability.
  • Details : Includes flows ran (with custom parameter indication), user email, timestamp, and before-after states.
  • For single datapoint edit, the event will include the datapoint value before and after edit, however
  • For range or multi edits, the audit events will show details about the datapoints count and time range for the edits

Datasource Access

  • Purpose : Monitor data access for security and compliance.
  • Details : Includes datasource accessed, user email, and timestamp.

User Whitelist

  • Purpose : Ensure proper tracking of user access changes.
  • Details : Includes user email, timestamp, and namespace.

Group Change

  • Purpose : Maintain accurate records of group membership changes.
  • Details : Includes group involved, user email, and timestamp.

Permission Change

  • Purpose : Audit changes to group permissions for security.
  • Details : Includes group, changed permissions list, and timestamp.

Manual Ingestion

  • Purpose : Track manual data ingestion for compliance.
  • Details : Includes flow design, user email, and timestamp.

Testing Framework Package Availability​

EWXDEV-8657

For partners using the BDD testing framework to test their business process implementation, the package pytest-bdd-ewx is now available on the package hosting accessible to partners. Contact your project consultant for access details.

API Request Improvements​

EWXDEV-9242

API requests have been enhanced to make the console load faster. Authentication requests accompanying all API calls are cached on page, resulting in a smoother user experience in the datasource view.

Preview Feature - Computed Flags​

We've received feedback regarding the 'glance-ability' of datasource details. We're soft-launching our computed flags feature, which are live-calculated flags visualised without being stored.

The datasource view has been enhanced with a new data point status highlighting feature. This preview functionality allows users to quickly visualise whether a data point has a "good" or "bad" status, represented by green or red points respectively on the chart. Additionally, the status flags have been added as a new column in the table view, enabling easy filtering and identification of important data points.

The status can be configured in the channel classifier view using a flexible structure that supports multiple conditions with logical operators. Users can set conditions based on data point values, attributes, and a range of comparison operators (==, >=, <=, <, >, !=, ). Both "AND" and "OR" operations are supported, providing granular control over the status criteria.

Feature highlights:

  • Quickly identify problematic data points with color-coded indicators (green for "good", red for "bad").
  • Visual cues on chart and table update in real-time.
  • Configure status conditions using a flexible structure supporting logical operators (AND, OR).
  • Visual cues on charts make data interpretation straightforward, updated in real-time.
    Flag Conditions
  • Tailored insights with channel-specific flag toggling using timeseries and datapoint attributes

This feature supports adding visualisation based on conditions evaluating timeseries on a single datapoint and its Datapoint Attributes value.

How can I test this feature?

  • Pre-requisite: Enable new datasource view deployment and beta features.
  • Configure the computed flags in APIv2’s channel classifier. Only API configuration is supported in this release
  • Once the configuration conditions has been set, you can head to datasource details and see your computed flags in action

Note: this feature is experimental and computed flags can be hidden unless the condition is removed from the datasource view.

Under the hood​

Continuous Vertical Pod Autoscaler (VPA)​

EWXDEV-9662

This release introduces a Continuous Vertical Pod Autoscaler (VPA) that automatically monitors and adjusts pod resource allocation. The VPA checks for pod restarts caused by resource exhaustion and scales up CPU and memory resources to prevent future OutOfCPU or OutOfMemory conditions. By dynamically optimising resources, the VPA improves pod reliability, increases application availability, and reduces operational overhead.

Audit Event BigQuery Writer​

EWXDEV-10192: Improved the speed of writing audit events to BigQuery using new writers powered by Go.

Defect Fixes

Channels Visibility Preferences​

Blacklisted Channels Hidden by Default​

EWXDEV-8236, EWXDEV-9599

When a user sets a channel as a blacklisted channel in the user settings, the blacklisted channels are now hidden by default from the chart and its legend, as well as from the table in the datasource view.

Default Visible Channels​

EWXDEV-9250: When a user sets a channel as a default visible channel in the user settings, these channels are now shown by default in the chart and table in the datasource view.

Audit Event Details Update​

EWXDEV-9389: The file path subject is no longer shown in the audit event, and the flow_config_id is removed if it has no value assigned.

Streaming Ingest Service​

EWXDEV-10012: Fixed an issue where the streaming ingest service didn’t process messages correctly.

API v2 and New Datasource View​

Preventing Duplicate Entities

EWXDEV-9505: Creating a new entity in APIv2 will no longer overwrite an existing entity with the same name. The expected response is '409 Conflict' for new services like Process Monitoring, however existing services still return ‘500’ at the moment.

Pagination Issue Fixed

EWXDEV-9679: Fixed an issue on the new channel classifier view where pagination didn't load all data available in the platform.

Look n' Feel

  • EWXDEV-9773: Added date filters for datapoints in the datasource view.
  • EWXDEV-9844: Moved datasource properties to the top of the datasource view for better visibility.
  • EWXDEV-9846: Cleaned up datasource view tabs, setting tags as the default tab and caching the API response.

Other Notable Updates

  • EWXDEV-8548: Removed unused datasource table from the advanced search view under schema tab.
  • EWXDEV-9923: Addressed GKE node scaling issue.
  • EWXDEV-9926: Fixed contract payload processing issue in EDM.
  • EWXDEV-10085: Resolved excessive GET requests issue when switching consoles.
  • EWXDEV-10372: Fixed issue with excessive requests upon logout.
  • EWXDEV-10477: Restricted editing of attributes to the appropriate channels in the frontend.
  • EWXDEV-10478: Implemented advanced-search API call for EQL validation.
  • EWXDEV-9754: Improved merge_datasources functionality in ewx-common.
  • EWXDEV-10227: Added flag configuration to the Channel Classifiers via APIv2 for visualising "bad" values in the data.

Impact Analysis

Infrastructure Changes
- We’ve completed migrations of major API services from AppEngine to Kubernetes. This transition is seamless from a customer perspective as all routing has been adjusted accordingly.

- Audit Events Writer: Rewritten in Go for improved performance and faster visibility of new audit events.
- Continuous Vertical Pod Autoscaler (VPA): Introduced VPA to automatically monitor and adjust pod resource allocation, enhancing pod reliability, application availability, and reducing operational overhead.

API Breaking Changes
- APIv2 Datapoint Attributes: Whole float values are serialized without the additional .0 (e.g., 5 instead of 5.0).
- APIv2 and APIv1.5 Channel Classifier: Name is now case-insensitive; use display name for case sensitivity.

Compliance
- Audit Event Retention: Audit events will be retained for 1 year before being compressed and archived. But will always be available via Cloud Storage

24.02 Hotfix 1

Better Compliance Visibility:​

  • Added audit events for various user actions, including:
    • Datasource access
    • User whitelisting/removal from a namespace
    • User addition/removal from a group
    • Group permission changes
    • Manual data ingestion
  • Enhanced audit events for group changes to include a list of changed permissions instead of a full permission dump.
  • Excluded detailed state changes from the "Datapoint Edit" action to avoid cluttering audit logs, relying on the edit history saved in Bigtable.
  • EWXDEV-10993: [original console] Audit events are now logged after a datapoint edit with the option to run a flow is selected.

What’s Changed:​

  • EWXDEV-8080:[original console] Fixed an inconsistent error message in the File Manager that appeared and asked for re-authorisation
  • EWXDEV-9057:[original console] “Load channels” button will be set to disabled once all the channels are loaded and no more channels can be loaded
  • EWXDEV-9491:[original console] Tag properties with type timestamp are now displayed with correct timezone in edit mode
  • EWXDEV-10177:[original console] Fixed a visual issue where the legend filter showed "No results found" event when results are returned
  • EWXDEV-10508:[original console] Editing a datapoint from continuous flow generates a new version as expected, see full impact analysis below for more details
  • EWXDEV-10549: [original console] Users can now select blacklisted channels to load them even if hidden at first load
  • EWXDEV-10627: [original console] Rule parameters of type timestamp default to the datasource timezone instead of New York timezone
  • EWXDEV-10677: [original console] Task results limit is now respected in the task listing results when set higher than 20 results per page
  • EWXDEV-11106: [original console] Using double quotes for queries in advanced search are now supported consistently with the API usage
  • EWXDEV-10757: [original console] Removed “Readonly” filter on the file manager search.

Impact Analysis:​

Continuous Flow Editing Behaviour Change​

What's Changing:

Area Affected: Timeseries versioning and access within continuous flows. Secondary impact on external data access methods relying on flow metadata to access continuous flows.

Summary of Change: We're enhancing how manual data edits are handled in continuous flows. Edits will now create new data versions instead of overwriting the existing one, in-line with how continuous datapoint updates work in the rest of the platform.

Why It Matters:

Impact on Your Usage:

Performance: No impact

Scalability: No impact

Resource Utilisation: No impact

Cost Change: No impact

Technical Explanation:

This change aligns continuous flow editing behaviour with our documented behaviour. It ensures data consistency and allows for faster loading of datapoint in a continuous flow

Additional Notes:

Important Considerations:

Users accessing continuous flow data outside the platform via flow metadata will find this data inconsistent for all datapoints edits in a continuous flow. This access method is not supported.

The correct way to access continuous flow data is by requesting the latest available timestamp for the desired data point.

24.02 Hotfix 2​

We've realised a degradation in processing performance in 24.02 and this hotfix is tackling this issue first and foremost, here's a small analysis of what has changed:

We've changed our message deduplication using pod-name in the logic; we streamlined the logic into a fast access using locking mechanism, and removed additional inefficient logic.

Other Improvements

  • Original Datasource view shows the display name for a channel instead of the technical name when provided by user.
  • Dashboard Widget QuickSetting - Fields are resized for better user input experience
  • Rule Framework - Datapoint values in dataframe can now be empty. Even if there are no Datapoint values, continue the processing pipeline will continue to process and handle the Datapoint attributes.

24.02 Hotfix 3​

Under the hood improvements mainly improving workloads that were stuck, and the robustness in writing data entities in atomic manner.

This hotfix also updates the Audit Event log message when uploading SOAP files, so that the filename is now shown as well.

24.02 Hotfix 4​

Rules Framework - Chain Rules Update​

Chain rules (including bulk, condition, and chain types) have been enhanced with new temporal control capabilities through the addition of two optional parameters: start_datetime and end_datetime.

New Parameters​

  • Start Datetime
    • Controls the beginning of the time range for triggered flows
    • Optional parameter
    • Type: datetime
  • End Datetime
    • Controls the end of the time range for triggered flows
    • Optional parameter
    • Type: datetime

Default Behaviour​

If datetime parameters are not specified, the system will:

  1. First attempt to use existing dataframe indexes
  2. If indexes are empty, default to the last two years

Technical Implementation​

Required API Configuration

Partners must update their rules configs with the following parameters:

    {
      "name": "start_datetime",
      "displayName": "Start Datetime",
      "widgetType": "DatePicker",
      "valueType": "datetime",
      "defaultValue": "",
      "optional": true,
      "description": "Start datetime for triggered flow"
    },
    {
      "name": "end_datetime",
      "displayName": "End Datetime",
      "widgetType": "DatePicker",
      "valueType": "datetime",
      "defaultValue": "",
      "optional": true,
      "description": "End datetime for triggered flow"
    }

Partner Actions Required​

No Action Needed

API Caching Fix​

Impact: Resolved caching issues affecting resource management for:

  • Namespaces
  • Users
  • Billing accounts

This resulted in user rights inconsistencies for users working with multiple namespaces

Resolution: Fixed issues with resource fetching and saving operations, improving data consistency and reliability.

Other Changes​

  • Fixed caching issue preventing namespaces from being correctly shown after being created initially.

24.02 Hotfix 5​

🔒 Important Infrastructure Update​

As part of our commitment to continuous improvement, this release includes critical infrastructure upgrades to our deployment management platform. Getting this update ensures your environment is prepared for upcoming feature releases and improvements.

UI Improvements​

  • Fixed a display issue affecting whitelisted channel visualisation when using French language

Stability & Performance Improvements​

  • Resolved an issue in handling existing file names when storing files from flows.
  • Resolved an issue that affected resilience in the cloud-notification-router workload, the impact affected 1 in 5 million processes, and resulted in the data not being stored
  • Improved retry mechanisms for temporary cloud service interruptions

Deployment Note​

The first time this fix is deployed to a customer environment it will recreate all pubsub subscriptions that support labelled flows. This means that all data contained in them will be lost. For this change to be deployed production environments must be completely shut down and the queues need to be empty before proceeding.