Flow Orchestration Overview
Flows are the platform's data-processing pipelines: configurable sequences of Python rules that validate, estimate, transform, calculate and forward data. This page explains how the pieces fit together; each piece has its own detailed documentation.
The building blocks
- A Flow Design is a pipeline made of one or more sequences, executed in order. Each sequence contains one or more rules, and later sequences can use the results of earlier ones — for example, values annotated by validation rules in sequence 1 are corrected by estimation rules in sequence 2.
- A Rule is a piece of Python code that operates on the data flowing through — validating, estimating, calculating, forecasting, or calling out to other platform services. Rules are configured with parameters via Rule configurations, so their behaviour is adjusted per flow without code changes.
- Flows come in two types: scenario flows produce independent versioned results per run (e.g. forecasts), while continuous flows build on existing data. See Types of flows.
How flows are started
A flow can be started in five ways:
- By ingestion — data arriving through any ingestion path runs the flows configured for it.
- On a schedule — a Trigger Schedule starts flows or aggregations on a cron expression.
- By another flow — flows chain: a parent flow triggers a child flow, optionally choosing the
child at runtime with a Decision Tree
(
conditional_chain_flow). See chain options. - By correlated messages — the Collective Trigger accumulates messages from upstream flows by correlation ID and starts a downstream flow when its condition is met (message count, deadline, or inter-message timeout).
- Manually — start a flow on demand from several pages in the console (for example the Datasources View) or via the API.
A note on the word "trigger"
Three distinct concepts share the word:
| Term | What it does |
|---|---|
| Trigger Schedule | Starts a process at a scheduled time (cron) |
| Collective Trigger | Starts a flow when correlated messages have accumulated |
| Chained flow trigger | A rule inside a flow starting another flow |
Monitoring
Flow executions can be tracked in Process Monitoring — per-process KPIs, statuses and alerting — once a process monitoring configuration is set up for the flow. In addition, each datasource shows its flow output and audit events in the Datasources View.