Skip to main content

Release Notes: Version 25.02

Discover the latest features and improvements in Version 25.02 for an enhanced user experience.

What's new​

API v2 Enhancements:​

We have made several enhancements to our API v2 to give you more control and visibility:

  • Start Flows with API v2: You can now start flows directly through our API v2.
  • Flow Metadata Search: A new endpoint has been added for searching flow metadata, and it now returns flow configuration information.

Rule Framework:​

The rule framework has also been enhanced with new capabilities:

  • Custom Logging from Rules: It is now possible to create audit events directly from your Transform Rules, by simply using the self.rule_logger in a transform rule.
  • Split Step Monitoring: Rules in your flows can now retrieve the amount of messages that were generated during the split step in the ingestion process. Please refer to the documentation for details on this new feature in the market_adapter_split_count section.

Data Management:​

Here is a list of improvements related to data management:

  • Advanced Search Feedback: Advanced search now provides more details if your query is not valid.
  • Timeseries Support: The platform now ensures microsecond support for timeseries indices. Please note that flows and ingestions will continue to support millisecond precision at this time.
  • Process Monitoring: We have added new "In Progress" metrics to Process Monitoring, allowing you to easily track the number of process instances that have started but not yet finished.
  • Deprecated Client Removed: The deprecated ewx-client has been removed in this version.

What's Available in the New UI:​

The following functionalities are available in the new UI, which can be used by enabling beta features:

  • Searching for datasources
  • Managing channel classifiers (and underlying channel families)
  • Creating market adapters
  • Managing live flags and process monitoring
  • Viewing audit events (Note: Links still lead to the classic UI for now)
  • Using enerGPT, our in-house AI assistant

Integration tooling​

We've made major improvements to our integration tooling, with a special focus on making your development and testing workflows smoother.

Enhanced Pytest Plugin:​

Our pytest-bdd-ewx plugin is now more powerful, allowing you to easily test a wider range of API v2 endpoints. You can now:

  • Validate Datasource Tags: Check and verify tags and their attributes on a datasource, just as you do for API v1.5.
  • Create Tags: Create tags with specific valid_from and version fields directly from your tests.
  • Verify Channel Attributes: Easily confirm the number of datapoints, attributes, and annotations for a channel in the Timeseries API v2 response, with support for filtering by date range, channel family, and more.
  • Validate Search Results: Ensure that your datasource search returns only the specific datasources you are looking for.
  • Monitor Live Flags: Get a live flag by name and use it to filter timeseries data, verifying the amount of flags per channel.

A full list of these new steps is available here

ewx-cli API v2 Support:​

We've added initial support for API v2 in ewx-cli, starting with Channel Families. To use this new functionality, you'll need to update your ewx-cli credentials. We've added a new add-apiv2-credentials command to simplify this process for existing projects.

  • Note: To use this new functionality, API v2 credentials must be created by an admin and added to your configuration.
  • Backwards Compatibility: The ewx-cli won't fail if a Channel Family configuration is missing, but it will show a warning to help you identify missing configurations.

Notable Improvements & Fixes​

We've addressed several issues and introduced key fixes across the platform for a more stable and intuitive experience.

Platform & Console​

  • Improved Search & Visibility: Advanced Search now provides more details on invalid queries, and the results table now displays all columns correctly.
  • More Accurate Data: We've resolved an issue where older annotations would appear on newer datapoints. Also, timeseries data now has microsecond precision, and a bug was fixed where flow timestamps were incorrectly set to zero.
  • User Experience: Linked the datasource flow in the TC to the flow configuration page, and fixed an issue where Datasource classifiers could not be changed more than once.
  • Access Control: All users now have access to the EnerGPT page, and audit events are now properly filtered by severity.

API v2​

  • More Accurate Filters: The Timeseries API now successfully filters on a specific version instead of just on ranges. We've also added support for comma-separated channel classifiers in API v1.5 timeseries calls.
  • Improved Validations: Case-insensitive checks for duplicate names have been added, and we no longer allow symbols for 'non-enum' attributes, improving data integrity.

Data Management & Rule Framework​

  • New Channel Family Naming: Channel Family names prefixed with EWX_ are no longer allowed to prevent conflicts between internal and customer-specific families.
  • JSON Market Adapter: We fixed a bug in the JSON MA that prevented multidimensional JSON files from being ingested.
  • Transform Rules: The get_existing_datasources method now returns an object for each datasource ID, including None for non-existing ones. Custom transform rules can now also be configured for a tag's valid_from field.
  • Data Integrity: We've added validation to check for proper ISO datetime values for a tag's valid_from and version attributes.

ML Serving​

  • More Cost-Effective ML Models: Our ML models are now more cost-effective when in an idle state, helping you optimize resource usage.

BREAKING CHANGE​

EQL queries: dropped support for aliasing of datasource_id table attribute in EQL. internal_datasource_id must now be used instead of datasource_id in EQL queries. This affects dashboard and advanced search queries.

25.02.01​

Bug Fixes​

  • Updated search to allow for more "fuzziness"
  • An issue was caused by an unexpected error in the code, which halted execution. All data ingestion that triggers crunch flows should now complete successfully.
  • Added retry mechanism in collect and adapt MA for google TransportError exception, caused by NewConnectionError.
  • Resolved an issue where the BigQuery ingest trigger schedule did not work due to a recent architectural change.

25.02.02​

Documentation

The Knowledge Base has been reintegrated directly into the platform and reorganized to provide clearer structure and easier navigation.

Bug Fixes​

  • Fixed issue that prevented replacing all occurrences of a query string in a query.
  • Added a bypass_decision_tree parameter to the send_collective_trigger method and CrunchTrigger object, enabling callers to skip the evaluation of any associated decision tree. This allows for conditional execution, where a decision tree may be defined but not executed based on external logic. The default behavior remains unchanged.
  • Increased autoscaler resources

25.02.03​

Bug Fixes​

  • Removed Billing account page from UI
  • Fixed an issue that prevented "Created On" attribute from showing for Live Flags
  • Added retry mechanism in collect and adapt MA for google TransportError exception, caused by NewConnectionError.
  • Fixed an issue that was preventing Audit Events from Collective Trigger pipeline to be written to BigQuery.
  • Fix an issue that was preventing to list Channel Families
  • Fixed all/enabled/disabled buttons for flow design page
  • Added retry error in run flow method for google Abort exceptions.
  • Added missing permissions for Channel Families and Live Flag features. To keep viewing or managing these resource please ensure that the user groups have this permission added. These permissions are no longer managed separately from the User Groups in the Original UI. Note: Live Flags is categorised under the Computed Flags permission category.
  • Fixed issue where Channels without DPAs could not be updated in the new UI

25.02.04​

Bug Fixes​

  • Resolved an issue that prevented datasource view in the NUI from loading causing a permission denied error
  • Resolved an issue where live flags using Enum DPAs would not evaluate properly when loading in a datasource
  • Resolved an issue that prevented users from deleting Live Flag configs in the NUI

25.02.05​

Bug Fixes​

  • Resolved an issue that prevented datasources with a dash in the ID from showing in the NUI

25.02.06​

Bug Fixes​

  • ewx-cli documentation is corrected for deprecated commands
  • Resolved multiple issues with permissions in the new API that did not respect the permissions of the user through their groups.

25.02.07​

Bug Fixes​

  • Resolved issue that prevented ingestions from processing due to an issue with creating audit events from transform rules.
  • The following issue was resolved: with the removal of the prepare pipeline in 25.02 (a part of the data processing pipelines), a bug was introduced that caused all messages to be routed to the processing pipelines of a single billing account. All messages that were processed still ended up in the correct tables, there were no issues in data separation on database level.

25.02.08​

Bug Fixes​

  • Resolved issue that added an invalid suffix to some EQL table names that resulted in Table Not Found errors.

25.02.09​

Bug Fixes​

  • Lists (in addition to dictionaries) are now able to be used in flow properties.

25.02.10​

Bug Fixes​

  • Improved Transformation Configuration: Adding a transform rule to a static field in version and valid from in Channels and Tags is now supported.

  • Improved Permissions Fixed issue when trying to retrieve timeseries for permissions with conditions.

  • Improved Audit Event Search: Fixed bug that prevented search by subjects (ie: flow id) in the audit events from the Datasource Details page

25.02.11​

Bug Fixes​

  • We now use a set of fallback machine types, so that if one runs out there's probably another. This reduces the risk of workloads not being scheduled to near zero.

25.02.12​

Bug Fixes​

  • Improved Market adapter properties serialization: Fixed issue when trying to serialize Market adapter properties.

25.02.13​

Bug Fixes​

  • Improved Timeseries Service: Fix issue when trying to set namespace in timeseries service.
  • Improved XML File Creation: Fix issue when trying to create XML response files. This issue was caused by the mechanism to remove duplicated elements and attributes.

25.02.14​

Bug Fixes​

  • Send bypass_decision_tree to CrunchTrigger model correctly.
  • Fixed issue in Avro schema in the detectors writers.

25.02.15​

Bug Fixes​

  • Improved BQ Streaming Service: Fixed a connection error in the BQ streaming process.

25.02.16​

Changes​

  • Disabled EnerGPT: EnerGPT is no longer available. If you need to enable it, please reach out to the Service Desk to do it.

25.02.17​

Bug Fixes​

  • Improved Timeseries Service: Fixed a bug where Datapoint Attributes of type 'timestamp' were being converted to naive timestamp objects, instead of remaining timezone-aware.

25.02.18​

Bug Fixes​

  • Fixed Frontend bug: Fixed an issue regarding the google analytics API, that was causing the Frontend Build to fail