Release Notes: Version 24.04

Discover the latest updates and improvements in Version 24.03 through these comprehensive release notes.

What's New

Major Features

1. EnerGPT Preview

We're excited to introduce EnerGPT, our new AI-powered assistant designed to help you build and configure the Energyworx platform more efficiently. This preview release brings intelligent assistance to streamline your configuration workflow.

Key Features:

• Interactive platform knowledge assistance. Try asking: when do I choose a continuous flow over a scenario one?
• Configuration and rules guidance. Try asking: how can I store my Datapoint Attributes in a rule while removing empty entries
• Release notes exploration. Try asking: what were the highlights of 24.03 release in terms of timeseries management
• Natural language interaction with platform documentation

Benefits:

• Reduced configuration time
• Faster access to platform knowledge
• Immediate answers to platform-related questions

Preview Phase Details:

• Available in the new console view (beta features must be enabled)
• Token limit: 5,000 tokens per user during preview
• Focused on configuration assistance and platform knowledge

How to Get Started:

• Enable beta features in your user settings
• Access EnerGPT through the new console view
• Start asking questions about platform configuration, rules, or recent release notes

Note: As this is a preview feature, we encourage feedback to help shape its development. The current token limit is designed to help us evaluate usage patterns and ensure optimal performance. You can send your feedback directly via email to communication@energyworx.org, or create a change request ticket through our portal.

2. Channel Family Retention Management

We're introducing comprehensive data lifecycle management through Channel Families, enabling automated enforcement of retention rules. This feature helps organizations optimize storage costs and maintain compliance requirements.

Key capabilities:

• Configurable Retention Policies

  • Version count-based retention
  • Time period-based retention
  • Combination rules (either limit can trigger retention)
  • Default 10-year retention policy for existing classifiers

• Automated Policy Enforcement

  • Google garbage collection integration
  • Up to 5-day completion window for cleanup
  • Automatic version management

• User Interface Integration

  • New console UI support
  • Channel Family management interface
  • Visual policy configuration

Implementation Notes:

• Channel Family assignments are permanent due to storage architecture
• Channel Family technical names should be no longer than 5 characters
• It's recommended to keep the number of channel families in your design less than 10 for most use cases. if you feel that you require more than 10 channel families, please reach out to us (either via support or directly to your Energyworx contact) and we are happy to work with you on the best practice implementation.
• Changes require Service Desk support
• All existing Channel Classifiers start with default retention

3. Business Process Monitoring Alerting - Get alerted when your business process has an issue

We're adding alerting capabilities to the Process Monitoring feature. Now the users can be notified by email when a metric of their choice matches given criteria. Key capabilities:

• Support for the number of successful and the number of failed instances metrics (more to be added in future updates).
• Convenient way of defining criteria by selecting a metric, operator and providing a threshold value to be checked.
• Support for multiple timezones to match user workflow.
• Ability to specify multiple emails.

How to use

To set up a notification the user provides a condition to be checked (a threshold value and an operator).

Email can take up to 5 minutes to be sent out once the metric values are matching the condition. The user provides a list of email addresses where the notification supposed to be sent.

4. Task Creation in Rules

We've introduced the ability to create tasks programmatically within rules, offering more flexibility in workflow automation:

Capabilities:

• Direct Task Creation
  • Use self.create_task within rules
  • No pre-configuration needed in flow design
  • Automatic subject relationship maintenance
• Comprehensive Property Control
  • Title and description
  • Task type and status
  • Priority levels
  • Assignee management
  • Task board assignments
  • Custom tags
  • Aggregation configurations
• Integration Features
  • Idempotent task creation
  • Automatic source tracking
  • Flow metadata linking

Example Usage:

self.create_task(title="Investigate High Usage", 
                 task_type="validation", 
                 status="new", 
                 priority=TaskPriorityEnum.HIGH, 
                 assignee_email="analyst@company.com", 
                 description="Usage exceeds threshold by 50%", 
                 taskboards=[123])

We suggest using this over configuring task in flow designs as users are limited to one per flow.

5. Enhanced Permission-Based Navigation

The platform now features a dynamic navigation menu that adapts to user permissions:

• Dynamic Menu Rendering
  • Automatic permission-based visibility
  • Empty category hiding

Important Notes:

• The menu automatically updates if permissions change
• Administrative functions are no longer organization-restricted
• All menu items correspond to specific permission sets
• Empty categories are hidden to reduce interface clutter

6. Search Improvements

We've upgraded the Search Datasources view with several user experience enhancements;

Long headers now gracefully truncate with tooltips, all timestamp displays are converted to local time with clear timezone indicators, and tag cell values can be copied with a single click. Additionally, we've added a quick-access button to the original datasource view while maintaining all existing search functionality.

Technical Improvements

1. Performance Enhancements

Infrastructure:

• Increased stability in Kubernetes scaling mechanisms
• Improved entity writer with transactional commits

API and Caching:

• Enhanced API caching mechanisms
• Query Optimisation
  • Improved data retrieval patterns
  • Enhanced filtering capabilities
  • Better handling of large datasets

2. Security Updates

Authorisation:

• Open Policy Agent Implementation: We're working on revamping the authorisation model in APIv2 to allow for faster and more flexible security compliance and policy management, in this release we laid out the technical foundation on the API.

Audit and Compliance:

• We have updated audit event log when Uploading SOAP file, so filenames are included now

3. Configuration Management

BigQuery ingest trigger:

For power users using ewx-cli for repeatable deployments; String replacement is available for the BigQuery ingest trigger schedule. For this, a new property 'bq_template_mapping' was added. This property should be of format {'{KEY}': 'Value'} where it will replace any instance of '{KEY}' in the BigQuery query by 'Value'. This will be done automatically for {PROJECT_ID} and {NAMESPACE}.

Chain Rules

Chain rules (bulk, conditional,chain) now have two new parameters, start and end date. If there are not given, the triggered flow will either use the dataframe indexes, if not empty, or the last two years. When deploying this fix for the first time to a new project, the rules configs need to be updated, using the api calls, by the support engineer doing the deployment:

{
"name": "start_datetime",
"displayName": "Start Datetime",
"widgetType": "DatePicker",
"valueType": "datetime",
"defaultValue": "",
"optional": true,
"description": "Start datetime for triggered flow"
},
{
"name": "end_datetime",
"displayName": "End Datetime",
"widgetType": "DatePicker",
"valueType": "datetime",
"defaultValue": "",
"optional": true,
"description": "End datetime for triggered flow"
}

Additionally, it's possible to use now comma-separated string for multiple datasource and flow configuration ids for bulk_chain_flow and conditional_chain_flow rules.

Advanced Search

The flow_stats table is a new table that contains information on the duration of every rule in a flow. It is available in advanced search. Note that before this feature can be used, the namespace needs to be bootstrapped. More info on how to use this table can be found on our documentation page.

Deprecation Notices

Tag Retrieval Function Deprecation

• Deprecated: get_tag function
• Replaced by three new functions:
  • get_latest_tag_version
  • get_active_tag_versions
  • get_all_tag_versions
• Timeline:
  • Deprecation: 24.04

API Changes

New Endpoints on APIv2

Channel Families API

• Retention management endpoints
• Configuration endpoints

Policy Management

• Role management endpoints
• Permission configuration endpoints
• Policy enforcement endpoints

Modified Endpoints

• Channel Classifiers
  • New field for assigning a channel family to a channel classifier

User Interface Updates

New Features

1. Search Enhancement

   • Introduction of a new datasource search view

   • Enhanced filtering capabilities

2. New Datasource View

   • ****Show/Hide All buttons are now visible by default in the datasource legends UI, removed from the Advanced section. Buttons maintain existing functionality and styling while being more accessible to users.
   • In response to partner feedback, we've expanded the chart zoom capabilities in the New UI to provide more precise and intuitive time range controls. These new options complement the existing zoom levels to offer greater flexibility in data visualisation. New Zoom Options:
     • Current Month: Shows data from the 1st of the current month up to the present moment
     • Last Month: Displays the complete previous calendar month
     • Last 7 Days: Provides a rolling weekly view

3. Audit Events View

   • Audit Events has a new checkbox that allows users to search through audit event properties through the UI. This could be used for example to search through rule logs. Note that this may have a performance impact on search results.
   • Trigger schedules will report errors in audit events when certain issues occur:
     • Technical name not existing
     • Wrong trigger schedule version
     • Incorrect values of parameters
     • Missing mandatory parameters

UX Improvements

• Dashboards: Set the column width in result tables on the Dashboard to auto instead of fixed sizes.

• Dashboards: Fixes an issue where visual components in the dashboard quick settings would overlap under extreme browser zoom and resize conditions. The fix ensures:

  • Proper component spacing is maintained at all zoom levels
  • Time frame editor remains fully usable during window resizing
  • Quick settings panel maintains correct layout regardless of browser settings

• New Datasource View: Tag rows wrap longer strings instead of scrolling

• When a flow tries to import a rule that doesn't exist an audit event with error message is shown.

Notable Fixes

Data Management

1. Timeseries Management
   • Fixed an issue where deleting a timeslice group only removed its latest version. Now, deletion properly removes all versions of a timeslice group with the same name, preventing unexpected version restoration when reusing names.
   • We've improved how Datapoint Attributes (DPAs) are displayed in the datatable view, making it easier to access and understand attribute information while maintaining existing functionality. New Features:
     • Consolidated Attributes Column: New dedicated column showing all attributes for each channel
       • Grouped presentation of attributes per channel
       • Highlighted styling for better visibility
       • Can be hidden when not needed
       • Existing Views Maintained:
     • Existing Views Maintained:
       • Individual attribute columns remain available
       • Per-attribute filtering and sorting capabilities preserved
       • Original column-per-attribute view continues to function as before
     • Channel Pivot Improvements:
       • All attributes automatically group when pivoting on a channel
       • Annotations summary remains available during channel pivot
       • Maintains existing annotation functionality
       • Quick access to complete attribute set without additional clicks
   • Enhanced timezone support for the new datasource view
   • Fixed an issue where missing preferred timezone settings triggered a non-fatal error in the Original UI
   • Corrected unexpected conversion of DPA timestamp timezones. Please Note: Any attributes with timestamp values need to be converted to Datetime_utc in the transformation configuration rules.
   • Resolved a bug where accessing a non-existent property in user settings caused unexpected behaviour
   • Annotations can now contain Infinity as a value.
   • Empty values show up as false and values that cannot be set to false in datasource table
   • It is possible to use dynamic channels with different Datapoint Attributes that are described in the Transformation Configuration. However, if there is a missing Datapoint Attributes, it won't raise any error.
2. Channel Management
   • Better dynamic channel handling
   • Improved DPA support
   • Enhanced classifier management
3. Process Handling
   • Resolved an issue in handling existing file names when storing files from flows.
   • CSV files with one column are now detected correctly in Transform Configuration view.
   • We've enhanced XML file handling in Transformation Configurations by adding automatic column detection capability. This improvement streamlines the configuration process for XML data sources.
   • This addition makes working with XML files as straightforward as working with other file formats in Transformation Configurations, reducing setup time and potential configuration errors.
   • When in a Transform Configuration, the datasource classifier has an unknown value, an error message will be raised, specifying it is an error in the datasource classifier and not the channel classifier
   • Market Adapter Import - Enhanced Error Reporting Fixed an issue with Market Adapter imports by adding audit event errors when imports fail, providing clearer visibility into import failures and making troubleshooting more straightforward for both users and support teams. The change ensures that:
     • Failed Market Adapter imports now generate specific audit events
     • Error details are captured in the audit trail
     • Users receive clear feedback when imports don't succeed
     • Improved trigger scheduling
     • Better error messaging
4. Environment
   • Fixed an issue where bootstrapping new namespaces failed with a 404 error due to missing billing account association. The system now properly sets the namespace_id on the billing account during namespace bootstrapping step.

Looking Ahead

Upcoming Changes

1. View Deprecations
   • From 25.01, we're setting the following new views as defaults as they achieve feature parity for end users:
     • New Datasource Search View
     • New Audit Events View
     • New Channel Classifiers Management View
2. Future Enhancements
   • We're working hard on improving on our search performance in 25.01
   • APIv2 Timeseries, we're achieving x4 improvements for querying timeseries. For more, reach out to Energyworx Support Team

Migration Recommendations

• Update to new tag retrieval functions in the Rule Framework

Known Issues

UI Inconsistencies

1. New datasource view: Datapoint information under charts doesn't match clicked datapoint, so it may see incorrect data when analysing charts
2. Performance Variation in New Datasource View: While the new datasource view generally offers improved performance, some users may experience slower timeseries data loading times than expected. We are actively investigating to optimise performance across all use cases.
3. New datasource view: Switching between charts and table view resets edits in table view

Channel Classifier and Channel Family Management

1. Symbol Datapoint Attributes values displayed as "[object Object]" instead of actual value
2. UI lacks validation for invalid or empty input for Datapoint Attributes names
3. Channel family with negative or 0s retention policy can still be created
4. Reserved channel family name "ewx" shouldn't be allowed to be used, however it's now possible.
5. Cancelling the creation of a Channel Classifier leads users to incorrect page
6. API v2 doesn't check for duplicate channel family technical names
7. On APIv1.5: a. Unable to set retention policy for channel families in API v1.5 b. APIv2 GET Channel Family details endpoint doesn't support channel families created via API v1.5 c. We recommend using APIv2 for managing channel families

Minor Releases

Minor Release 1

• Fix the data loss that is caused by cloud-notification-router workload

Minor Release 2

Bug Fixes

• Enabled/fixed SOAP endpoints in APIv1.5 and ensured SOAP functionality is routed via APIv1.5.
• Moves the chart datapoint click event to the rendered point instead of the chart object for details rendering consistency
• Changing deadline looper ctp limit to 1000 for E-redes envs.
• Fixed issue with datapoint edits incorrectly using continuous mode when datasource flows are setup.
• Fixed issue with edits not being loaded on timeseries data for datasource flows when no source channel data is present.
• Finds the proper datapoint clicked on the Datasource chart view
• Adds option to remove the empty value in a Dropdown component
• In channel family's creation page, the interface will only allow users to submit their request after all validation warnings have been resolved
• Fixed an issue where Datapoint Attributes with symbols type were visualised as 'object' instead of their proper value.
• Fixes an issue in channel family creation page where the display name was used for the technical name as well.
• Fixed collective trigger application packaging

Improvements

• Enhanced pagination for APIv2 timeseries queries
• Optimised resource allocation for datasource retrieval

Benefits

• Reduced number of API requests for multi-channel data
• More efficient data retrieval for large datasets
• Improved system resource utilisation

Minor release 3

Bug Fixes

• Task descriptions rendering are no longer limited to 1k characters
• Adding retry in prepare source data logic
• Fixed issue when creating/updating rules. Now we check the status code of the response, and return the correct error message based on the response data.
• This includes a fix for dealing with the error messages when creating a new flow or tc, and also when updating a tc. We now check the status code of the BE request and based on it and the response data, we return the correct error message
• Added "No tags found" message when tag list contains only "group" tag.
• Added "ModuleNotFoundError" exception for csv files where the body starts with the letter "c" and contains more than one row.
• Namespaces now have an optional "sub_region" field that will override the "region" where a more precise location can be supplied for the BigQuery data. Values like "europe-west1" or "us-central1" are supported. For the full list of supported values, check this page: https://cloud.google.com/compute/docs/regions-zones#available. Please make sure to specify the region, not the zone ("europe-west1", not "europe-west1-b")

Improvements:

• Users can now access our knowledge base from inside the platform
• Formats month(s) the same as version(s) for Channel Family retention policy
• Improved API Pod Autoscaling based on the RPS (requests per second) coming to the API.
• Reduced the CPU and Memory resource usage of the API Workload.
• Reduced the minimum Number of Pods used by the API Workload.
• Updated the GKE version to be a minimum of 1.31.
• Added APIv1.5 scaling based on Google Cloud Load Balancer metric, to reduce the idle cost of the API workloads and to make the scaling faster.
  • We've implemented advanced scaling capabilities for API v1.5 using Google Cloud Load Balancer metrics, optimising both cost efficiency and performance during varying workload conditions.
  • Cost Change Impacts:
    • Significant reduction in idle workload costs
    • More efficient resource allocation during low-traffic periods
    • Optimised cost-to-performance ratio
  • Scalability
    • Faster scaling response during peak load periods
    • More responsive adjustment to sudden traffic changes
    • Improved handling of high-concurrency scenarios

Note: Changes may be less noticeable during light usage

Minor release 4

Bug Fixes

• Fixed an issue with the writer component that affected how some tags were written into storage, that affected the flow metadata endpoint where retrieval of property values were missing in some cases

Minor release 5

Bug Fixes

• Allow the rule parameter to be changed without the source code (until now, the rule parameter could only be changed if the source code was changed)
• Added more extensive API validation for channel families
• Consistently returns 404 when no channel families are present.
• Added API validation for channel families
• The form for adding tag on multiple datasources was changed because it didn't contain the property type and besides that, it was defined in controller. The type of tags was also changed, it should be sent as a list (it was defined as an object and I changed it to be a list).
• Created on date on file manager is showing correct date, Created on date am and pm is shown as expected
• Fixed an error in the labelling of AND and OR in Channel Families on the NUI.
• Removed the part that grants viewers access to energyworx users from create_new_project_in_google.sh
• Editing data points value from the chart works as expected.
• A graph widget query startdate is based on UTC

Minor release 6

Bug Fixes

• Uncaught exceptions for particular combinations of uploaded files content's are dealt properly now and the file is properly processed.

Minor release 7

Bug Fixes

• Original Datasource view - Fixes an issue where editing a datapoint with '0' value, removed the datapoint from the editing view
• Fixing restoring historical versions of flow configurations for datasource flows
• Enabled/fixed SOAP endpoints in APIv1.5 and ensured SOAP functionality is routed via APIv1.5.
• Dashboard graph widget rendering as expected.
• It is possible to search only a value and not a list

Minor release 8

Bug Fixes

• New tag versions were created when the TC configuration was set to non-SCD. Now only one version remains.

Minor release 9

Bug Fixes

• Flow results visualization missing backwards compatibility for some existing flow executions.
• Making sure get_existing_datasources returns an object for each datasource id (including None for non existing datasource ids).
• Correction of database instance location in processing pipeline.

Minor release 10

Bug Fixes

• Fixed issue that prevented flow metadata being written to BigQuery.

Minor release 11

Bug Fixes

• Added retry mechanism for google RefreshError Exception, in the collect Market adapter logic.

Minor release 12

Bug Fixes

• Now it is possible to filter legend channels by their displaynames
• Channel Properties are now returned by the tag search endpoint (datasource/search) in APIv1.

Minor release 13

Bug Fixes

• Now the time series data time zone matches the dashboard graph's x-axis, which shows UTC time.

Minor release 15

Bug Fixes

• Custom transform rules can now be configured for a tag's valid_from field in the Transformation Configuration.
• Added retry for pubsub exeception in market adapt logic
• Fixed the remove and update tags for multiple datasources

Minor release 16

Bug Fixes

• Custom transform rules can now be configured for a tag's valid_from field in the Transformation Configuration.
• Storing the tags once when the file is ingested multiple times.

Minor release 17

Bug Fixes

• Longer task descriptions can be generated. Previous maximum size was 1500 chararcters.
• Creation of files from flows has gotten more robust.
• Add Service Account IAM to topic, that allows the Google Storage bucket to send a notification to timeseries aggregation topic.

Minor release 18

Bug Fixes

• Added a retry mechanism to resolve data adaptation failures caused by network exceptions.
• Resolved an issue that prevented the bigquery_ingest trigger from functioning without a Market Adapter assigned.
• Updated AG Grid Key.