Skip to main content

Release Notes: Version 24.03

What's New

1. EQL Improvements - Speed and Data Safety​

Breaking Change​

So if your eql query has no set insertion_date or _PARTITIONTIME parameter, and you’re querying on September 3rd, 2024, the results will only include data inserted after September 3rd, 2023. However, the data within those results can still refer to older dates (for example, if you’re backtracking to calculate billing for a past period).

If the date is specified, then the behaviour is the same as before.

The namespace needs to be bootstrapped for queries without a insertion_date or _PARTITIONTIME parameter to work.

Benefits:

  1. Increased querying speed
  2. Enhanced data safety
  3. Expected decrease in unintentional BigQuery cost and usage

Change in default behavior:

EQL queries without a specified insertion_time or _PARTITIONTIME will now only include data inserted within one year from the query date by default.
Example of new behaviour:

If an existing eql statement is ran on September 3rd, 2024, without specifying insertion_time or _PARTITIONTIME, results will only include data inserted after September 3rd, 2023.
However, the data within these results can still refer to older dates (e.g., for backtracking or historical calculations).

Consistency with previous behaviour:

If insertion_time or _PARTITIONTIME is explicitly specified, the behavior remains the same as before.

Increase querying speed and add data safety, the platform now exposes insertion_time for all tables available to eql. An additional benefit is an expected decrease in unintentional BigQuery cost and usage

The change in behaviour is that now eql queries without a specified insertion_time or _PARTITIONTIME will only include data that has been inserted within one year from the date of the query by default.

2. Flow Duration Limit Warning​

We're introducing a new behaviour to help our partners build more efficient and robust flows. This update adds warnings for flows that exceed certain duration thresholds, promoting better performance and resource utilisation.

New Behaviour​

  • 5-Minute Warning An audit event is generated when a flow's execution exceeds 5 minutes.
  • 10-Minute Warning An audit event is generated when a flow's execution exceeds 10 minutes.
    In a future release, this will become an error event, and flows exceeding 10 minutes will be automatically aborted.

Benefits

  • Early identification of performance issues
  • Opportunity to optimise business rules
  • Improved overall system efficiency

Technical Details

Duration Measurement:

  • Ingestion and ingestion flows: Measured collectively
  • Chained flows: Measured individually

Looking Ahead
These warnings are designed to help partners optimise their flows proactively. By addressing performance issues early, partners can ensure their processes achieve business goals efficiently and avoid future disruptions when stricter limits are implemented.

We encourage all partners to review and optimise any flows that trigger these new warning events.

3. New Units available in Channel Classifiers​

We've expanded the unit options available in channel classifiers:

  • Gas Datasource Type: Cubic Meter per Hour (m3/hour)
  • Water Datasource Type: Litre per Minute (L/min)

4. Knowledge Base Access​

  • Users can now access the Energyworx Knowledge Base directly from within the platform.
  • No additional login or authentication required.
  • Simply click on the "Knowledge Base" item in the side menu to open it in a new tab.

5. Platform Version Visibility​

  • The current platform version is now visible within the console.
  • Users can easily reference the deployed version by looking at the bottom of the side menu.

6. New Alpha Datasource Search View​

  • We're developing a refreshed search page with a focus on simplicity and reduced clutter.
  • An alpha sneak peek of this feature is available to users with the 'beta features' flag enabled.
  • This new view represents our ongoing efforts to enhance user experience and streamline data access.

7. New Datasource View Improvements​

Edit Reason Requirement​

To enhance security and compliance, providing an 'edit reason' is now mandatory when:

  • Editing a datapoint value
  • Modifying annotations
  • Changing Datapoint Attributes

This change ensures a clear audit trail for all data modifications in the new datasource view.

Note: We've renamed 'value' to 'datapoint value' for improved clarity.

Annotations and Datapoint Attributes Enhancements​

  • Both zero '0' and 'null' values are now inspectable from the details panel.
  • Users can click on datapoints and their annotations from the chart when the annotation is activated.

Tags Improvements​

We've made several updates to improve the tags functionality:

  1. Layout Optimisation:
    • Up to 4 tags are now displayed in one row, maximising screen usage for power users.
    • Tags are expanded by default for increased glanceability.
  2. Simplified View:
    • The 'Versions' button is only visible when more than one version is available for a tag.
    • The 'Show Removed' button appears only when 'removed' tags are available.
  3. Enhanced Styling:
    • Both tags' 'version' and 'valid from' are styled distinctly from other tag properties.
  4. Future-Dated Tags:
    • Tags with a 'valid from' date set in the future are now hidden by default.

8. Process Monitoring Improvements​

We're excited to announce several enhancements to our Process Monitoring feature, aimed at improving clarity, usability, and error feedback:

Metrics Display Optimisation​

  • The metrics results row has been redesigned for better space utilisation.
  • Descriptions are now hidden behind tooltips, allowing more room for key metrics.
  • Time zone information has been added to Start and End time metrics.
  • For increased clarity, we've renamed the "Start Time" metric to "First process instance start time".

Real-Time Data Access​

  • A new "refresh" button has been added to the metrics display.
  • This feature ensures users can access the most up-to-date metrics at any time.

Improved Error Feedback​

  • We've enhanced the system's response to non-existent flow configuration IDs in Transformation Configurations.
  • When a user enters an invalid ID, the system now generates an audit event with a Warning level.
  • This improvement provides clear and immediate notification of configuration issues.

Unshipped in this release

As part of our ongoing commitment to user-centric design and streamlined functionality, we've made several changes to remove or modify features that weren't meeting their intended purposes. This section outlines these changes and the reasoning behind them.

1. Transport Adapter​

  • What : The transport adapter has been removed from the platform and isn't accessible from the console's menu.
  • Why : It didn't provide the integration resilience we had aimed for.
  • Alternative : We've developed a new component offering improved integration with external systems and hubs.
  • Next Steps : For integration needs, please contact our service desk for an enhanced solution.

2. Advanced Search Panel Cleanup​

  • What : Unused tables 'annotations' and 'annotations_bt' have been removed from the advanced search panel.
  • Why : To simplify and streamline the search experience. We saw no active usage of these tables.

3. 'Generated ID' Helper Column​

  • What : The 'Generated ID' helper column in the transformation configuration view is no longer available as an option for use with flow properties.
  • Why : To reduce complexity and focus on more frequently used options.

4. User Settings - Channel Features​

  • What : We've consolidated channel features into two main options: 'blacklisted channels' and 'Default visible channels'.
  • Why : To simplify user choices and improve overall user experience.
  • Removed : The "whitelisted channels" feature has been removed.

Notable Fixes

1. Task Management​

  • Fixed an issue where task boards with no filter didn't load correctly.
  • Task results limit is now respected in the task listing when set higher than 20 results per page.
  • Resolved a layout issue in the Dashboard view affecting autoscaling of visual elements.
  • Advanced Search now consistently supports using double quotes for queries, aligning with API usage.

3. Datasource Views​

Original Datasource View​

  • Fixed an issue where editing the same datapoint multiple times without refreshing led to duplicated entries in the datatable.
  • Resolved the problem of Datapoint Attributes with the same name being duplicated in all matching columns.
  • Fixed the audit event stats display.
  • Datasource view now shows the display name for a channel instead of the technical name when available.
  • Addressed an issue where editing a datapoint with '0' value removed it from the editing view.
  • Improved functionality to allow loading of Blacklisted Channels when needed.
  • Fixed an issue where clicking 'switch to channel view' didn't behave as expected.
  • Resolved an issue with edits and the pivot column causing user filters to reset.

New Datasource View​

  • Fixed an issue where trying to open a datasource via the new console led to the channel classifiers page.

4. User Experience​

  • Resolved an issue where users had to log out twice from the console (from both original and new views).
  • Fixed an issue where updating the blacklisted channel list from the new user settings view didn't take effect.
  • The new login view's popup prompting the user to login is now shown in the expected location.

5. Rule Framework​

  • Datapoint values in dataframe can now be empty. The processing pipeline will continue to handle Datapoint attributes even if there are no Datapoint values.

6. Technical Improvements​

  • Fixed an issue to prevent the simultaneous use of classifiers and channel_family in timeseries service. An exception is now thrown if both parameters are used. The filter by channel_family only has been fixed.

Minor Releases

Minor Release 2​

  • When choosing default visible channels, only those channels are visible on the chart.

Minor Release 3​

  • Fixed issue where old channel flow rules would fail to load in ingestions/flows.

Minor Release 4​

  • Additional warnings on audit events when a flow is triggered from a transformation configuration, even when no flow configuration is assigned. This now results in a warning Audit Event to the user about the Flow Config missing.

Minor Release 5​

New Features and Improvements​

SOAP Integration Enhancements​

  • Enabled SOAP response endpoint for improved integration capabilities
  • Added channel properties and information completeness in responses
  • Improved validation in store rules to verify DPA existence in channels

User Interface Improvements​

  • Users with (billingaccount create, update, read) permission set can whitelist users for a namespace from a different billing account.
  • We've re-enabled Channel Classifiers' new view after taking some time to work on issues that affected the user experience.
  • When viewing a limited number of channels on chart on the datasource view, the user is able to see the channel name next to the line chart.
  • We've reduced the number of visible markers when viewing a large number of datapoints on chart for better user experience
  • Updated UI messaging for ad blocker extensions to use "whitelisting" terminology
  • Added ability to delete assignee from task details
  • Improved search functionality to prevent immediate result reset when modifying search terms
  • Tags with future 'valid from' dates are now hidden by default

Data Processing and Performance​

  • Fixed timezone handling for Datapoint Attributes timestamps. When storing Datapoint Attributes of type timestamp, the value needs to be converted to Datetime utc in transformation configuration rules
  • Enhanced retries for failed timeseries loading operations in the Rules framework
  • Improved file creation from flows robustness.
  • Added cache expiration to channel classifier in timeseries endpoint on APIv2
  • Improved handling of dynamic channels with different DPAs in Transformation Configuration

Chart and Visualisation Updates​

  • Improved gap visualisation in new datasource view's chart.
  • Enhanced new datasource view to show channel display names instead of technical names
  • Timezones on the new datasource detail view uses user's timezone consistently
  • Show/Hide All buttons are directly visible in datasource legends panel.

Bug Fixes​

  • Fixed process monitoring metrics loading in process configuration
  • Fixed an issue where bootstrapping new namespaces failed with a 404 error due to missing billing account association. The system now properly sets the namespace_id on the billing account during namespace bootstrapping step.
  • Addressed logout synchronization between original and new consoles
  • Fixed an issue where messages were processed with the default workload instead of the one corresponding to the label of the process

Technical Improvements​

  • Enhanced API handling for flows without flow configurations
  • Added Decision Tree ID to audit events for better traceability

Minor Release 6​

Platform Improvements​

We've expanded our audit event search capabilities. Users can now search through audit event properties and rule logs using a new checkbox option on the Audit Events page. While this feature provides deeper insights into your audit trail, please note that enabling this enhanced search functionality may impact performance.

Consistent Timezone Experience​

We've expanded timezone consistency across the platform. The system now respects user timezone settings in the new views for Channel Classifiers and datasource view, and the File Manager, ensuring a more coherent date display experience throughout the platform.

New Datasource View Enhancements​

Multiple improvements have been made to the handling of Datapoint Attributes (DPA). The system now correctly displays and enables editing of boolean "true" values, while also properly managing empty boolean values in the table view. We've also enhanced DPA value rendering in tables through improved channel key matching.

Misc.​

  • Dashboard - The recipe listing functionality on Dashboards has been restored to full operation.
  • Timeslice Groups - We've also resolved an issue where Timeslice Groups would enter a persistent loading state, preventing updates and deletions.
  • Fixed a small issue affecting channel classifiers for French language users.
  • We've standardised the handling of None and NaN values in annotation properties to prevent data inconsistencies.
  • The platform now demonstrates improved resilience during high-volume ingestion scenarios, with enhanced retry mechanisms for cloud service interruptions.

Technical Improvements​

The platform's core infrastructure has been strengthened with improved error handling for cloud operations and enhanced data consistency across components. Various UI and visualisation issues have been addressed to provide a more stable user experience.

Minor Release 7​

Knowledge Base Integration

  • Users can now access the Energyworx Knowledge Base directly from within the platform
  • No additional login required
  • Available through the side menu, opens in a new tab

Namespace Regional Options Enhancement

Namespaces now have an optional “sub_region“ property field that will override the “region“ where a more precise location can be supplied for the BigQuery data.

Values like “europe-west1“ or “us-central1“ are supported. For the full list of supported values, check this page: GCP region zones

Please make sure to specify the region, not the zone \(“europe-west1“, not “europe-west1-b“\)

Improvements​

Infrastructure & Performance

  • Enhanced API Pod Autoscaling based on requests per second (RPS)
  • Optimized CPU and Memory resource usage for API workload
  • Reduced minimum number of API workload pods
  • Updated GKE version to minimum 1.31

Task Management

  • Removed 1k character limit on task descriptions
  • Improved task description display capabilities

Bug Fixes​

Datasource View & Data Management

  • Fixed datapoint edits incorrectly using continuous mode with datasource flows
  • Resolved issue with loading datapoints edits on timeseries data for datasource flows
  • Fixed visualisation of Datapoint Attributes with symbols type
  • Corrected datapoint panel updates when changing clicked channels
  • Improved chart datapoint click event handling for consistent details rendering

Flow & Rule Management

  • Improved error handling when creating/updating rules with clearer messages
  • Fixed issues with transformation configuration (TC) error messaging to be more user-friendly
  • Fixed collective trigger application packaging
  • Adjusted deadline looper CTP limit to 1000 for large customers' environments

User Interface

  • Fixed an issue where Datapoint Attributes with symbols type were visualised as 'object' instead of their proper value.
  • Fixed datapoint info accuracy under charts to match clicked points