Rule Based Message Validations

Rule Structure Example

Each validation rule is carefully crafted to ensure data integrity. For instance:

{ "header_rules": [{ "mapped_element": "correlation_id", "validation": { "type": "equals", "mapped_field": "correlation_id", "error_code": 999, "error_msg": "CorrelationID komt niet overeen" } }, ], "rules": [{ "element": "MarketEvaluationPoint/mRID", "element_type": "int", "validation": { "type": "valid_ean18", "value": null, "error_code": 650, "error_msg": "EAN-code-18 is niet valide." } },{ "element": "DateAndOrTime/startDateTime,DateAndOrTime/endDateTime", "element_type": "date,date", "validation": { "type": "start_before_end", "value": null, "error_code": 661, "error_msg": "Ongeldige tijdperiode." } } ] }

header_rules

• mapped_element : Points to an element from the mapper that is part of the header_maps.
• validation.type : Denotes the validation function to be applied.
• validation.mapperd_field : Points to an element from the mapper that is part of the field_maps.
• validation.error_code and error_msg : Define the error handling in case of validation failure.

rules

• element : Identifies the XML path to the data point to be validated. If the rule needs multiple inputs, separate the input elements with a comma in one string like “a/b,c/d”
• element_type : Specifies the expected data format (e.g., list, string, date). If there are multiple inputs, the types must match the inputs. E.g. “string,list”
• validation.type : Denotes the validation function to be applied.
• validation.value : Provides a value for comparison, where necessary.
• validation.error_code and error_msg : Define the error handling in case of validation failure.

These configurations are structured as followed in the relevant GCP bucket:

• SOAP service rules are in the bucket rules/<msg_type>_<msg_type_id>.json
• InREST service rules are in the bucket rules_msgspec/<msg_type>.json

For outgoing messages, the msg_type is set as an attribute on the pubsub message.

The message map configuration configures the msg_type_id in the maps with the process_type_id.

{ ... "field_maps": { "process_type_id": { "function": "get_from_message", "args": "ProcessTypeID" }, ... } ... }

The available rules are at the end of this document. Additional rules can be incorporated as required to accommodate evolving validation needs.

External Access to Configuration

For stakeholders needing access to the integration bucket outside the Energyworx environment:

1. WinSCP is the recommended tool for secure file transfer.
2. Upon setting up WinSCP, the following configurations are necessary:
3. Select the S3 protocol under a new site configuration.
4. Enter storage.googleapis.com as the Host name.
5. Input the provided Access key ID and Secret access key. (generate one for the client)
6. Specify the directory as <gcp-project>-integration-service.
7. Save these settings and initiate the connection using the Login button.

Available Validation Rules

1. valid_ean18

Description : Checks if an input is a valid EAN-18 code.

Arguments :

input1: The EAN-18 code to validate

Example Configuration :

{ "element": "MarketEvaluationPoint/mRID", "element_type": "int", "validation": { "type": "valid_ean18", "value": null, "error_code": 650, "error_msg": "EAN-code-18 is niet valide. } }

2. valid_ean13

Description: Checks if an input is a valid EAN-13 code.

Arguments:

input1: The EAN-13 code to validate

Example Configuration:

{ "element": "Product/barcode", "element_type": "int", "validation": { "type": "valid_ean13", "value": null, "error_code": 651, "error_msg": "Invalid EAN-13 code." } }

3. equals

Description: Checks if the input equals a specified value.

Arguments:

input1: The value to check

value: The value to compare against

Example Configuration:

{ "element": "Status/code", "element_type": "string", "validation": { "type": "equals", "value": "ACTIVE", "error_code": 652, "error_msg": "Status must be ACTIVE." } }

4. sequence_starts_with

Description: Checks if the first value of a list equals a given value.

Arguments:

input1: The list to check

value: The expected first value

Example Configuration:

{ "element": "Sequence/items", "element_type": "list", "validation": { "type": "sequence_starts_with", "value": "START", "error_code": 653, "error_msg": "Sequence must start with START." } }

5. same_length

Description: Checks if the length of two lists is equal.

Arguments:

input1: The first list

input2: The second list

Example Configuration:

{ "element": "Data/keys,Data/values", "element_type": "list,list", "validation": { "type": "same_length", "value": null, "error_code": 654, "error_msg": "Keys and values must have the same length." } }

5.1 not_empty_same_length (JSON ONLY)

Description: Overloads same_length to first check if one of the lists is not empty.

6. resolution_points_ratio

Description: Checks if the number of points equals the expected number based on a given resolution within a current day.

Arguments:

input1: Start date (format: "YYYY-MM-DDTHH:MM:SSZ")

input2: End date (format: "YYYY-MM-DDTHH:MM:SSZ")

input3: List of data points

value: Resolution in minutes

Example Configuration:

{ "element": "TimeSeries/start,TimeSeries/end,TimeSeries/values", "element_type": "date,date,list", "validation": { “type” "resolution_points_ratio", "value": "15", "error_code": 655, "error_msg": "Incorrect number of data points for the given time range and resolution." } }

7. range

Description: Checks if the value of input1 is within a given range.

Arguments:

input1: The value or list of values to check

value: A comma-separated string or list of allowed values

Example Configuration:

{ "element": "Product/category", "element_type": "string", "validation": { "type": "range", "value": "Electronics,Clothing,Books", "error_code": 656, "error_msg": "Invalid product category." } }

8. range_in_past

Description: Checks if the given date range is in the past.

Arguments:

input1: Start date (format: "YYYY-MM-DDTHH:MM:SSZ")

input2: End date (format: "YYYY-MM-DDTHH:MM:SSZ")

Example Configuration:

{ "element": "HistoricalData/start,HistoricalData/end", "element_type": "date,date", "validation": { "type": "range_in_past", "value": null, "error_code": 657, "error_msg": "Date range must be in the past." } }

9. date_in_past

Description: Checks if a given date is in the past.

Arguments:

input1: The date to check (format: "YYYY-MM-DDTHH:MM:SSZ")

Example Configuration:

{ "element": "Event/occurrenceDate", "element_type": "date", "validation": { "type": "date_in_past", "value": null, "error_code": 658, "error_msg": "Event date must be in the past." } }

10. date_in_past_hours

Description: Checks if a given date is in the past by a specified number of hours.

Arguments:

input1: The date to check (format: "YYYY-MM-DDTHH:MM:SSZ")

value: Number of hours in the past

Example Configuration:

{ "element": "Transaction/timestamp", "element_type": "date", "validation": { "type": "date_in_past_hours", "value": "24", "error_code": 659, "error_msg": "Transaction must be within the last 24 hours." } }

11. gl_date_in_future

Description: Checks if a given date is in the future.

Arguments:

input1: The date to check (format: "YYYY-MM-DDTHH:MMZ")

Example Configuration:

{ "element": "Appointment/scheduledDate", "element_type": "date", "validation": { "type": "gl_date_in_future", "value": null, "error_code": 660, "error_msg": "Appointment date must be in the future." } }

12. start_before_end

Description: Checks if the start date is before the end date.

Arguments:

input1: Start date (format: "YYYY-MM-DDTHH:MM:SSZ")

input2: End date (format: "YYYY-MM-DDTHH:MM:SSZ")

Example Configuration:

{ "element": "DateAndOrTime/startDateTime,DateAndOrTime/endDateTime", "element_type": "date,date", "validation": { "type": "start_before_end", "value": null, "error_code": 661, "error_msg": "Invalid time period." } }

13. date_offset_equals

Description: Checks if the date offset is equal to a given value.

Arguments:

input1: Start date (format: "YYYY-MM-DDTHH:MM:SSZ")

input2: End date (format: "YYYY-MM-DDTHH:MM:SSZ")

value: Expected offset in days

Example Configuration:

{ "element": "Subscription/startDate,Subscription/endDate", "element_type": "date,date", "validation": { "type": "date_offset_equals", "value": "30", "error_code": 662, "error_msg": "Subscription period must be exactly 30 days." } }

14. date_offset_max_month

Description: Checks if the date offset is equal to or less than a given number of months.

Arguments:

input1: Start date (format: "YYYY-MM-DDTHH:MM:SSZ")

input2: End date (format: "YYYY-MM-DDTHH:MM:SSZ")

value: Maximum number of months

Example Configuration:

{ "element": "Contract/startDate,Contract/endDate", "element_type": "date,date", "validation": { "type": "date_offset_max_month", "value": "12", "error_code": 663, "error_msg": "Contract duration must not exceed 12 months." } }

15. date_offset_whole_days

Description: Checks if the date offset is a whole number of days.

Arguments:

input1: Start date (format: "YYYY-MM-DDTHH:MM:SSZ")

input2: End date (format: "YYYY-MM-DDTHH:MM:SSZ")

Example Configuration:

{ "element": "Reservation/checkIn,Reservation/checkOut", "element_type": "date,date", "validation": { "type": "date_offset_whole_days", "value": null, "error_code": 664, "error_msg": "Reservation must be for whole days." } }

16. sequence_order

Description: Checks if the values in a sequence are in the correct ascending order.

Arguments:

input1: The sequence to check

Example Configuration:

{ "element": "Process/steps", "element_type": "list", "validation": { "type": "sequence_order", "value": null, "error_code": 665, "error_msg": "Process steps must be in ascending order." } }

17. sequence_no_duplicates

Description: Checks if there are no duplicate values in a sequence.

Arguments:

input1: The sequence to check

Example Configuration:

{ "element": "Team/memberIds", "element_type": "list", "validation": { "type": "sequence_no_duplicates", "value": null, "error_code": 666, "error_msg": "Team member IDs must be unique." } }

18. all_present

Description: Checks if all the values in a given list are present in the input.

Arguments:

input1: The list to check

value: Comma-separated string or list of required values

Example Configuration:

{ "element": "Form/fields", "element_type": "list", "validation": { "type": "all_present", "value": "name,email,phone", "error_code": 667, "error_msg": "All required fields must be present." } }

19. only_if_present

Description: Checks if a specific value is present, no other values may be present in the input.

Arguments:

input1: The list to check

value: The specific value to allow

Example Configuration:

{ "element": "Order/specialInstructions", "element_type": "list", "validation": { "type": "only_if_present", "value": "RUSH", "error_code": 668, "error_msg": "Only RUSH is allowed as a special instruction." } }

20. all_present_once

Description: Checks if a combination of values is present only once.

Arguments:

input1: First list of values

input2: Second list of values

Example Configuration:

{ "element": "Inventory/productIds,Inventory/locations", "element_type": "list,list", "validation": { "type": "all_present_once", "value": null, "error_code": 669, "error_msg": "Each product must be assigned to a unique location." } }

21. decimal_precision

Description: Checks if the value is a decimal with the correct precision.

Arguments:

input1: The value or list of values to check

value: The required decimal precision

Example Configuration:

{ "element": "Product/price", "element_type": "float", "validation": { "type": "decimal_precision", "value": "2", "error_code": 670, "error_msg": "Product price must have exactly 2 decimal places." } }

22. max_decimal_precision

Description: Checks if the value is a decimal with a maximum precision.

Arguments:

input1: The value or list of values to check

value: The maximum allowed decimal precision

Example Configuration:

{ "element": "Measurement/value", "element_type": "float", "validation": { "type": "max_decimal_precision", "value": "3", "error_code": 671, "error_msg": "Measurement value must have 3 or fewer decimal places." } }

23. no_negative_values

Description: Checks if there are no negative values in the input.

Arguments:

input1: The value or list of values to check

Example Configuration:

{ "element": "Order/quantities", "element_type": "list", "validation": { "type": "no_negative_values", "value": null, "error_code": 672, "error_msg": "Order quantities cannot be negative." } }

24. in_combi_range

Description: Checks if combinations of two lists are in a given range.

Arguments:

input1: First list of values

input2: Second list of values

value: Comma-separated string of allowed combinations

Example Configuration:

{ "element": "Product/category,Product/subcategory", "element_type": "list,list", "validation": { "type": "in_combi_range", "value": "Electronics:Smartphone,Electronics:Laptop,Clothing:Shirt,Clothing:Pants", "error_code": 673, "error_msg": "Invalid product category and subcategory combination." } }

25. object_in_combi_range

Description: Checks if combinations of elements from an object are in a given range.

Arguments:

input1: The object to check

value: Comma-separated string of allowed combinations

namespace: The namespace to use for finding elements in the object

Example Configuration:

{ "element": "ValidationResult", "element_type": "object", "validation": { "type": "object_in_combi_range", "value": "Pass:Valid:None,Fail:Invalid:Repair,Fail:Invalid:Replace", "namespace": "result:", "error_code": 674, "error_msg": "Invalid combination of validation result, status, and action." } }

26. in_combi_range_multiple

Description: Checks if combinations of two lists are in a given range, allowing for multiple valid options for each key.

Arguments :

input1: The key to check

input2: The list of values to check against the allowed values for the key

value: A string representation of a dictionary where keys are the possible input1 values, and values are lists of allowed input2 values

Example Configuration :

{ "element": "Product/category,Product/tags", "element_type": "string,list", "validation": { "type": "in_combi_range_multiple", "value": "Electronics: ['smartphone', 'laptop', 'tablet'], Clothing: ['shirt', 'pants', 'dress']", "error_code": 675, "error_msg": "Invalid product category and tags combination." } }

27. not_empty

Description: Checks if all the values in the input are not empty or null.

Arguments:

input1: The value or list of values to check

Example Configuration:

{ "element": "User/name,User/email,User/phone", "element_type": "string,string,string", "validation": { "type": "not_empty", "value": null, "error_code": 676, "error_msg": "All user fields must be filled." } }

28. in_range_max_xdays_in_future

Description: Checks if a date is within a specified number of days in the future, if it's in a certain range.

Arguments:

input1: The value to check against the range

input2: The date to check (format: "YYYY-MM-DDTHH:MMZ")

value: A string in the format "range_value,max_days"

Example Configuration:

{ "element": "Appointment/type,Appointment/date", "element_type": "string,date", "validation": { "type": "in_range_max_xdays_in_future", "value": "URGENT,7", "error_code": 677, "error_msg": "Urgent appointments must be scheduled within the next 7 days." } }

29. in_range_in_complete_period

Description: Checks if the difference between two dates matches a specified period for a given key.

Arguments:

input1: The key to check

input2: Start date (format: "YYYY-MM-DDTHH:MM:SSZ")

input3: End date (format: "YYYY-MM-DDTHH:MM:SSZ")

value: A string of comma-separated items in the format "key:period:unit"

Example Configuration:

{ "element": "Subscription/type,Subscription/startDate,Subscription/endDate", "element_type": "string,date,date", "validation": { "type": "in_range_in_complete_period", "value": "MONTHLY:1:D,YEARLY:1:Y", "error_code": 678, "error_msg": "Subscription period does not match the specified type." } }

30. resolution_points_ratio_one_day

Description: Checks if the number of points equals the expected number based on a given resolution within a single day.

Arguments:

input1: Start date (format: "YYYY-MM-DDTHH:MM:SSZ")

input2: List of data points

value: Resolution in minutes

Example Configuration:

{ "element": "DailyData/date,DailyData/values", "element_type": "date,list", "validation": { "type": "resolution_points_ratio_one_day", "value": "15", "error_code": 679, "error_msg": "Incorrect number of data points for a single day with 15-minute resolution." } }

31. resolution_points_ratio_in_range

Description: Checks if the number of points equals the expected number based on a given resolution within a specified date range.

Arguments:

input1: Start date (format: "YYYY-MM-DDTHH:MMZ")

input2: End date (format: "YYYY-MM-DDTHH:MMZ")

input3: List of data points

value: Resolution in minutes

Example Configuration:

{ "element": "TimeSeriesData/start,TimeSeriesData/end,TimeSeriesData/values", "element_type": "date,date,list", "validation": { "type": "resolution_points_ratio_in_range", "value": "60", "error_code": 680, "error_msg": "Incorrect number of data points for the given time range with 1-hour resolution." } }

32. max_value

Description: Checks if all values are less than or equal to a specified maximum value.

Arguments:

input1: The value or list of values to check

value: The maximum allowed value

Example Configuration:

{ "element": "Order/itemQuantities", "element_type": "list", "validation": { "type": "max_value", "value": "100", "error_code": 681, "error_msg": "Individual item quantities cannot exceed 100 units per order." } }

33. date_start_time

Description : Checks if the time in a iso 8601 timestamp is equal to value.

Arguments:

input1: The element that holds the values to check

value: The allowed time value in '%H:%M:%S'

Example Configuration:

{ "element": "MeasurementSeries/PeriodStart", "validation": { "error_code": "661", "error_msg": "period does not start at 23:15:00", "type": "date_start_time", "value": "23:15:00" } }

34. date_offset_biggerthan_for_reason

Description : Checks the offset between allocation day and message creation based on reason code.

Arguments:

input1: Start date of the period (format: "YYYY-MM-DDTHH:MM:SSZ")

input2: creation date of the period (format: "YYYY-MM-DDTHH:MM:SSZ")

input3: the reason of the measurement

value: A mapping of reason:minimal_offset_in_day. So DAY5:1 means that for the reason of DAY5 there is a minimum offset of 1 days for the creation of the message. all means there is no limitation on the offset.

In this example: A message (DAY5, DAY10) can never be send before the actual measurement day (starting period) A message (DAY5, DAY10) can never be send ON the actual measurement day (starting period) a message with reason code MANUAL has no restriction

Example Configuration:

{ "element": "measurement_series.period_start,header.date_time,measurement_series.reason", "element_type": "date,date,string", "validation": { "error_code": "682", "error_msg": "Metering data is delivered outside window", "type": "date_offset_bigger_than_for_reason", "value": ["DAY5:1","DAY10:1","MANUAL:all"] } },

35. match_tag_property_to_message_field

Description : Check is the value of a certain datasource tag property is equal to a given value from the message. The check find all the values and call will call the check match_tag_property_to_value

Arguments:

input1: Meter_id / datasource_id (MarketEvaluationPoint/mRID)

input2: Allocation date of the message of the period (format: "YYYY-MM-DDTHH:MM:SSZ")

input3: field that holds the value to check aginst.

value: A mapping of tag_name:tag_property. So mv:ean means it wil find the tag mv, find the value for the property ean so it can be checked against the value of input3

Example Configuration:

{ "element": "measurement_series/MarketEvaluationPoint/mRID,measurement_series/DateAndOrTime/endDateTime,EDSNBusinessDocumentHeader/Source/SenderID", "element_type": "string,date,string", "validation": { "error_code": "655", "error_msg": "De MV is niet geregistreerd als MV op de aansluiting", "type": "match_tag_property_to_message_field", "value": "mv:ean" } },

36. match_tag_property_to_value

Description : check is the value of a certain datasource tag property is equal to a given value. the check will call the platform to find the tag and value

Arguments:

input1: Meter_id / datasource_id (MarketEvaluationPoint/mRID)

input2: Allocation date of the message of the period (format: "YYYY-MM-DDTHH:MM:SSZ")

value: A mapping of tag_name:tag_property:value. So mv:ean:8712423010284 means it wil find the tag mv, find the value for the property ean and checks if it is equal to the value 8712423010284

Example Configuration:

{ "element": "measurement_series/MarketEvaluationPoint/mRID,measurement_series/DateAndOrTime/endDateTime", "element_type": "string,date", "validation": { "error_code": "655", "error_msg": "De MV is niet geregistreerd als MV op de aansluiting", "type": "match_tag_property_to_value", "value": "mv:ean:8712423010284" } },

37. equals_msg_type_with_postfix

Description : checks is the messagetype combined with a postfix equals the value of a mapped field. This is created as a header_rule

Arguments:

input1: (mapped_element) content type for instance MEASUREMENT_SERIES_NOTIFICATION_N20

value: (mapped_field) process_type_id for instance N20

message_type: Will be filled based on the context

Example Configuration:

{ "mapped_element": "content_type", "validation": { "type": "equals_msg_type_with_postfix", "mapped_field": "process_type_id", "error_code": 999, "error_msg": "ContentType komt niet overeen" } },

38. Validate_uniqueness

Description : Check is the value or multiple values from a message is unique over all incomming messages. All values from the mapped elements are combined to a key. Redis is checked if that key already exists. If not it's unique and it's added to Redis for future checks. All keys get a default ttl of two weeks, but this can be configuerd per check.

Arguments:

input1..n: Values from the message. can be as manny as needed

config: First part is a key_prefix to add to the generated key to add extra config options, second is the ttl in seconds. 1209600 is two weeks. (so after two weeks the key gets deleted)

Example Configuration:

{ "element": "measurement_series/MarketEvaluationPoint/mRID,measurement_series/DateAndOrTime/endDateTime,EDSNBusinessDocumentHeader/Source/SenderID", "element_type": "string,date,string", "validation": { "error_code": "670", "error_msg": "Er is al eerder een bericht met dit kenmerk ontvangen.", "type": "validate_uniqueness", "config": "key_prefix:1209600" } },

This completes the documentation for all the validation rules in the provided code. Each rule can be used to validate different aspects of XML/JSON objects, providing flexible and powerful validation capabilities for various data structures and requirements.