Skip to main content

Release Notes: Version 23.03+

Here's a sneak peek at what's new:

Schema Optimisation: Data Point Attributes​

As Energyworx platform aims for cheaper and efficient implementation, the schema optimisation is an initiative that focuses on how timeseries is stored in our platform, this results in two big changes: Data Point Attributes & Channel Families.

Data Point Attributes

Users who have a lot of attributes that belong to timeseries datapoint can now make use of Datapoint attributes; the concept allows for storing data of the following types [bool, enum, float, timestamp] up to 80% more efficient storage. Datapoint Attributes' timeseries can be read and written from the same places annotations are.

This feature isn’t compatible with existing channel classifiers without a migration, but new channel classifiers support it out of the box. Currently, from the API

Channel Families

Users are able to set up timeseries data that will be access frequently together under the same channel family, for optimised data loading experience.

This feature isn’t compatible with existing channel classifiers without a migration, but new channel classifiers support it out of the box. Currently, from the API

Timeseries Aggregation​

AKA: Virtual Datasources on BigQuery

Continuing with the new vds_aggregation_ingest trigger schedule that we introduced in the previous release, we improved it in this release such that it does not need a Market Adapter to be configured any more. This means that the aggregation of the timeseries is automatically creating the datasource and channels of the datasource from the results of the aggregation.

Collective Trigger Pipeline​

Our replacement for bulk- and aggregation- dataflows feature

We introduced a new service called The Collective Trigger. This service collects messages from flows (triggering flows) in the platform and groups them by an identifier called the Correlation ID. This new service will replace the existing bulk trigger.

When a certain condition is reached (e.g. a number of messages collected per Correlation ID), then the service will handle the start of a flow (a result flow) processing with the arguments passed on the Collective Trigger messages.

These messages can be sent by a rule being executed in a flow by calling the function send_collective_trigger. This function can set two main types of parameters:

  • Flow execution parameters: the parameters for the flow execution, e.g. the flow design id, the flow properties, etc.
  • Collective Trigger handling: conditions on how to handle the messages for a certain Correlation ID, e.g.: the amount of expected messages, the deadline expected for the desired amount of messages, etc.

The flow properties of the messages received are combined per Correlation ID and shared with the result flow.

There are certain conditions that can be set for handling the Collective Trigger. They can set waiting windows for messages of the same Correlation ID in two different ways:

The deadline: a timeout window which starts after the first message is received and indicates that a specific amount of messages for the same Correlation ID is to be expected before this time is reached.

The timeout in between messages: a timeout window that starts with each message and indicates how long it is expected for a new message of the same Correlation ID to arrive.

When a condition of the above is expired and a message corresponding to that same Correlation ID is received, then we define these messages as Late Data. We can configure several possible behaviors to handle this Late Data:

  • Discard: simply discard the message.
  • Trigger flow combining all data again: trigger the message again with the combination of all flow properties.
  • Restart the window: restart the waiting window.

The following functions in the rule framework are affected by the implementation of this new feature/service:

  • New send_collective_trigger function is created, this can be used from any rule to call the Collective Trigger function.
  • send_bulk_trigger function is redirected to send_collective_trigger function.
  • Following send_bulk_trigger parameters have been set to defaults and will be deprecated:
  • number_of_current_items: set to 1
  • trigger_late_flow: set to True
  • send_crunch_aggregation_trigger is removed and any existing functionality has been migrated to send_collective_trigger.
  • send_bulk_trigger function is marked as deprecated and will be removed in the future release.

Deployment Note: during the deployment of this release all business processes that uses the bulk functionality should be stopped. The business process can be restarted after the deployment is finished.

APIv1.5 - Migration to Python3​

Python2 is no longer supported by Google starting the 1st of February 2024, therefore an upgrade to Python3 is needed. Below is the stacks of technologies we upgraded to or now use in this migration:

  • Python3.11
  • Flask-openapi3
  • Pydantic
  • Google cloud ndb (instead of Google appengine ndb)

In this migration, we created new API blueprints which includes specific endpoints definitions. All request/response models were recreated using Pydantic and the database models in the controller layer were decoupled by introducing a separate pydantic models for request and responses. Last, we mapped these models with the ndb models. to_model() method is used to map a pydantic model to an ndb model before sending it to the service; and from_model() method is used to map an ndb model returned by the service to a pydantic model, which will be the response of the API.

This new API blueprints can be found in api_py3/packages/energyworx-console/gae/ewx_gae/api/v1_5

Edited Datapoints Audit​

In this release, we added the possibility to verify the previous version of an edited datapoint, along with important information about who did the edit, when and what value they edited a datapoint to.

This information is visible on the data table and chart, additionally, when a user clicks on a datapoint, edit information will be visible in the datapoint panel in Datasource View.

Fixes in this release 🪲​

  • Fix an issue in timeseries aggregation where not all channels were created

  • EWXDEV-8549: Fixes an issue in timeseries aggregation where the Name and ID in the aggregation configuration weren’t always persisted

  • EWXDEV-7481: Fixes an issue in timeseries aggregation where channels with whitespaces in the column name weren’t created

  • EWXDEV-7663: Fixes an issue where similar tasks weren’t grouped by in the platform

  • EWXDEV-8225: Fixes an issue where changing the source channel for a channel flow before running it did not take effect

  • EWXDEV-8589: Fixes an issue where file tags where disappearing after manual ingestion

  • EWXDEV-8718: Fixes an issue where timeseries aggregation’s filter, the filter now supports using the same tag and property multiple times, enabling queries like tag.prop = A OR tag.prop = B

  • EWXDEV-8719: Fixes an edge case where uploading files with no default tags caused an issue with ingestion

  • EWXDEV-8734: Fixes an issue affecting the 'Run Flow' menu item from the flow metadata component.

  • EWXDEV-8741: Fixes an issue affecting the creation of tasks from a flow.

  • EWXDEV-8743: Fixes an issue in transformation configuration affecting dynamic channel classifiers with no display name

  • EWXDEV-8756: Fixes an issue where in some cases, a Bigtable DeadlineExceeded error would end up in an audit event instead of being retried.

  • EWXDEV-8757: Fixes an issue where timeseries aggregationtrigger didn’t create a Virtual Datasource when no chained flow was set

  • EWXDEV-8782: Fixes an issue where Virtual Datasources filters weren’t displayed in the console

  • EWXDEV-8780: Adds a daily timeout reset for collective trigger

  • EWXDEV-8786: Fixes an issue where timeseries aggregation chained flows weren’t triggering underlying datasources on first trigger.

  • EWXDEV-8811: Fixes an issue where running a scenario flow on edited datapoint breaks the datapoint edit API

  • EWXDEV-8817: Fixes an issue with rendering API Documentation page description

  • EWXDEV-8818: Fixes an issue affecting writing Detectors from flows that aren’t marked as readonly

  • EWXDEV-8826: Fixes an issue affecting “channel family” create endpoint

  • EWXDEV-8829: Fixes an issue where reloading a transformation configuration causes the page to be empty

  • EWXDEV-8853: Deleted datasource classifiers aren’t displayed by default on Datasource Classifier page on the console

  • EWXDEV-8854: Fixed an issue where flows with custom parameters failed to trigger

  • EWXDEV-8862: Fixes an issue with flows not loading on Datasource view

  • EWXDEV-8865: Fixes an issue affecting users running a flow from the task board

  • EWXDEV-8866: Fixes an issue affecting users trying to add a lane to existing task board

  • EWXDEV-8866: Fixes an issue affecting the status filter for task boards

  • EWXDEV-8875: Addressed a workaround while loading datasources by using channel view as default for datasources with more than 12 flows.

  • EWXDEV-8911: Resolved an issue preventing joining of tz-naive with tz-aware DatetimeIndex.

  • EWXDEV-8913: Fixed an issue where Virtual Datasource created via API had no data.

  • EWXDEV-8915: Resolved issues with loading transformation configurations with multiple channels in the console.

  • EWXDEV-8916: Fixed issues related to loading Timeseries in the console.

  • EWXDEV-8933: Standardised time formatting in Datasource Search to align with production standards.

  • EWXDEV-8935: Fixed issues related to range editing multiplying for datapoints.

  • EWXDEV-8936: Aligned the names for task creation attributes to resolve an error in GET/run/search.

  • EWXDEV-8937: Split channel classifier and timeseries GET requests to prevent query parameter overloads.

  • EWXDEV-8942: Addressed the issue of the missing priority field when creating tasks from a board page.

  • EWXDEV-8961: Corrected an issue in the initialisation process for creating legacy VDS.

  • EWXDEV-8962: Addressed the disappearance of the Load Channels button in the console when no flow timeseries is available.

  • EWXDEV-8969: Ensured Datapoint Attributes are correctly formatted and readable in the frontend.

  • EWXDEV-8970: Resolved a problem preventing the manual editing of boolean tag properties.

  • EWXDEV-8972: Fixed an issue causing annotations to not display in the frontend console when no timeseries is visible.

  • EWXDEV-8977: Modified the Channels tab to prevent triggering of datasource flows.

  • EWXDEV-8989: Resolved an issue preventing admin users from accessing trigger details.

  • EWXDEV-8995: Fixed an issue where conditionally chained datasource flows required a source channel.

  • EWXDEV-8998: Corrected a problem causing the defer-task-queue to pause automatically.

  • EWXDEV-9008: Addressed a UI issue affecting multiple flow selections in the console.

  • EWXDEV-9009: Resolved a UI issue with dependent flow checkboxes in the console.

  • EWXDEV-9014: Implemented a fix to enable unlinking tasks from a task board.

  • EWXDEV-9031: Fixed a store rule issue not storing DPAs with NaN or None values.

  • EWXDEV-9046: Addressed inconsistency in default channel view setting in the console.

  • EWXDEV-9056: Ensured persistent display of the channel/flow view switch in the console.

Platform Stability Enhancements

We've addressed several critical issues to improve platform reliability and performance:

  • Fixed DNS resolution failures for datastore operations
  • Resolved DLQ message handling for stream-crunch operations
  • Improved deadline handling for StoreData operations
  • Enhanced flow view editing capabilities

Configuration Updates

  • Updated E-REDES Monthly and Billing Scaling Parameters in infrastructure configuration

Fixes in this release 🪲

  • EWXDEV-11444: Resolved message handling in DLQ for ewx-edp-dso stream-crunch
  • EWXDEV-11639: Fixed flow view editing errors
  • EWXDEV-11980: Addressed datastore.googleapis.com DNS resolution failures
  • EWXDEV-12253: Improved deadline handling for data storage operations
  • EWXDEV-12154: Updated scaling parameters for E-REDES billing system

Deployment Note: Standard deployment procedures apply. No service interruption expected.

23.03 Hotfix 45​

Bug Fixes​

[EWXDEV-12205] CTP Limit Adjustment​

  • Changed deadline looper CTP limit to 1000 for certain environments, addressing CPES response status issues.

[EWXDEV-12253] Data Storage Timeout​

  • Added retry in prepare source data logic to handle "Deadline of 60.0s exceeded StoreData failed" errors.

[EWXDEV-12432] Data Retention​

  • Fixed data loss caused by cloud-notification-router workload, preventing datasource loss during high-volume ingestion.

Infrastructure Improvements​

[EWXDEV-12172] Cache Optimization​

  • Increased Redis instance size to 5GB for all instances (more cost-effective than 4GB configuration).

23.03 Hotfix 46​

Bug Fixes​

[EWXDEV-12253] Data Storage Reliability​

  • Added retry mechanism in prepare source data logic to address "Deadline of 60.0s exceeded StoreData failed" issues.

​

23.03 Hotfix 47​

Bug Fixes​

[EWXDEV-11230] SOAP Endpoint Integration​

  • Enabled/fixed SOAP endpoints in APIv1.5 and ensured SOAP functionality is routed via APIv1.5, resolving ingestion issues with customer-specific workflows.

[EWXDEV-12449] Market Adapter Processing​

  • Added retry mechanism for Google RefreshError Exception in the collect Market adapter logic, fixing issues with unprocessed customer's messages.

[EWXDEV-13276] Timeout Handling​

  • Added ServiceUnavailable retry in prepare_source_data.py to address 60.0s timeout issues.

Improvements​

[EWXDEV-8113] Flow Management Enhancement​

  • Introduced new way of stopping flows with the ability to persist flow data saved up to the point where the flow is stopped by user code.

23.03 Hotfix 48​

  • Certain partners have access to get_datasource function is available in decision trees now.
  • When all the requested data is present in the cache, return only the requested time window.
  • Channel Properties are now returned by the tag search endpoint \(datasource/search\) in APIv1.
  • More errors are retryable in the processing pipeline.

23.03 Hotfix 49​

  • The VEE processes resources have been increased for large-use-case environments.
  • Added retry for pubsub exeception in market adapter's logic

23.03 Hotfix 50​

  • Adjusted the soap outbound request timeout times and persisted changes to the soap outbound task queue.
  • Updated custom market adapters implementations for custom use cases

23.03 Hotfix 51​

  • The Datasource Billing writers resources have been increased for large-use-case environments.
  • Updated custom market adapters implementations for custom use cases

23.03 Hotfix 52​

SOAP endpoint creation now routes to the correct API backend service again. Creation of files from flows has gotten more robust. Added retry mechanism in collect and adapt MA for google TransportError exception, caused by NewConnectionError.

23.03 Hotfix 53​

Increased pod-custom-metrics-stackdriver-adapter resources

23.03 Hotfix 54​

Updated retry mechanism for google TransportError exception, caused by NewConnectionError.

23.03 Hotfix 55​

Allow Market Adapter Split messages to be sent to Storage when the message is greater than the Pubsub limit size. These messages will be processed by prepare pipeline normally.

23.03 Hotfix 56​

Fix Pubsub size message verification to allow the overhead of base64 encoding.

⚠️ Warning

Deprecation Notice​

We are in the process of deprecating ewx-client and would recommend using our public API for data access.