Trigger Schedule

A trigger schedule is used to run repeating processes in the platform. It is triggered at a certain time, based on the cron job that is set in its configurations.

Not to be confused with

A trigger schedule starts processes on a time schedule. It is distinct from the Collective Trigger, which starts a flow when correlated messages have accumulated, and from flows chained by other flows. See the Flow Orchestration Overview for how these fit together.

Common use cases:

• Running recurring aggregation
• Start a flow in multiple datasources
• Starting a time based process that has no incoming file to trigger it

Available triggers' technical names and versions are shown in the following table:

Technical name	Description
bigquery_ingest	Aggregates based on a custom BigQuery query
flow	Executes a flow
vds_aggregation_ingest	Aggregates based on a particular configuration
virtual_datasource	Aggregates based on a simple configuration

How to create Trigger Schedules with BigQuery ingest

For some business processes, running an automated Big Query is needed. For this, Energyworx has created the Trigger Schedule functionality. On this page, we will explain how to set it up, and how to easily connect it to a new Market Adapter and Transformation Configuration.

Firstly, make sure you are on the correct environment and namespace of the Energyworx platform. Then, navigate to Trigger Schedule, under the Smart Integration tab.

Then, click on Create in the upper-right of the window.

The “Create A Trigger” window pops up. In that window, we have sections called “Details”, “Trigger” and “Properties”. Firstly, we fill in the Details. For the Name, you can choose any descriptive name. The technical name, however, should be precisely “bigquery_ingest” such that the platform knows that it will be triggering a Big Query. Please fill in a good description and set the version to V1. Don't forget to set the Status to Enable, such that it is ran for testing.

Secondly, we fill in Trigger. This will be a CRON expression that makes it very easy to schedule automated processes based on UTC time. Don't worry if you don't know CRON expressions, the Console makes it very easy to create these CRON expressions.

Lastly, we fill in the Properties. The needed properties are “filename” which is used to save to file to the File Management, “market_adapter_id” which is used to ingest the file, “bigquery_query” which is the query that creates the file (in CSV or JSON format), and with “tags” you can set the tags (comma separated) to the created file, which will make it easier to find it back in the File Management. Optionally, you can specify "format" (either "CSV" or "JSON") to control the output format - if not specified, it defaults to CSV.

After successful triggering, a new file should be present in the File Manager, with a file name consisting of the trigger name and the trigger datetime. Furthermore, the set tags are present, making it easier to find all files coming from this trigger schedule. Lastly, the market_adapter_id is directly set and used for ingestion.

Of course, you might want to create a new a Transformation Configuration with a Market Adapter. If you're new to creating a Transformation Configuration, please follow the page Transformation Configuration , and then overwrite the market_adapter_id in the trigger schedule.

The following video shows how to create these triggers:

If you want to make your queries interchangeable between namespaces a helpful approach is to use template replacement. Your query can be used as a template where mappings for configurable fields in the bigquery query should be of format {'{KEY}': 'Value'} where it will replace any instance of '{KEY}' in query by 'Value'. You can specify these mappings in the parameter bq_template_mapping of your trigger. Protected keys are {PROJECT_ID} and {NAMESPACE} which will fill automatically the current project id and namespace.

Export Format Options

By default, BigQuery trigger schedules export data in CSV format. However, you can also export to JSON format by specifying the format property in your trigger configuration.

Supported formats:

• CSV (default) - Exports data as comma-separated values
• JSON - Exports data as JSON, with each row as a JSON object

The format should be specified as 'format': 'json' in your trigger properties. This is particularly useful when your data contains complex nested structures, or when an export contains data for multiple datasource and you need to ensure that the data for a single datasource is not split across multiple files.

How to create a Flow Trigger Schedule

A trigger schedule with the technical name flow runs a flow on a schedule. Set the technical name to precisely flow, version V1, and provide these properties:

Property	Description
flow_configuration_id	Required. The id of the flow configuration to run.
datasource_ids	The datasources to run the flow on, as a comma-separated list of ids (e.g. Site_A,Site_B). Required unless datasource_ids_query is given.
datasource_ids_query	Alternative to datasource_ids: a BigQuery SELECT whose first column is a datasource id. Provide either datasource_ids or datasource_ids_query, not both.
start_datetime / end_datetime	Optional. Bound the data window; default to all.
source_classifier	Optional. The source channel classifier to use.

Selecting datasources dynamically with a query

Instead of maintaining a fixed datasource_ids list, datasource_ids_query lets the trigger decide at run time which datasources to run the flow on. The query is executed each time the trigger fires and the flow is started on every datasource id it returns — so datasources that are added or removed later are picked up automatically, with no trigger change.

Write a query whose first selected column is the datasource id, referencing the datasources table:

SELECT id FROM `{PROJECT_ID}.{NAMESPACE}.datasources` WHERE name LIKE 'Kyos%'

• {PROJECT_ID} and {NAMESPACE} are substituted automatically for the trigger's own project and namespace, and table references are qualified to that namespace — so the same trigger works across namespaces.
• For any other placeholders, use the bq_template_mapping property (same format as the BigQuery ingest trigger): {'{KEY}': 'Value'}.
• If the query returns no rows, the trigger logs a message and does nothing that run (it does not fail).

Example properties for a trigger that validates every "Kyos" price datasource each morning:

Property	Value
flow_configuration_id	(id of your flow)
datasource_ids_query	SELECT id FROM `{PROJECT_ID}.{NAMESPACE}.datasources` WHERE name LIKE 'Kyos%'

How to start a trigger schedule from the API

A trigger schedule is generally started by a cron job, when it is scheduled to run. However, in some cases it is necessary to manually start a trigger schedule, for instance during testing. This is possible through the API, following the steps below:

1. Go to the Trigger Schedule page under the Smart Integration tab
2. Find the trigger schedule you want to run and copy the trigger id (this is different to the market adapter id). Make sure the trigger is set to Enabled. If it’s Disabled, executing the trigger will be rejected
3. Go to API Documentation page under the Support tab
4. Find the Trigger tab and look for the call POST /ewx/v1/trigger/execute
5. Click the right panel with example values to copy over the text to the body, and substitute the id with your copied id from step 2
6. Click try out - if successful you will get a 200 code as the response