Datasources

Search Datasources

Datasource Visibility in Search

Datasources are indexed for search in near real time, with one nuance that matters when you create them through the API:

• A datasource created with at least one tag is indexed immediately and appears in search right away.
• A datasource created without tags is not indexed until either a tag is added to it, which triggers indexing, or the periodic re-index runs.

So always create datasources with at least one tag. If none of your business tags apply yet, a small provenance tag does the job and documents where the datasource came from:

{
  "id": "METER_002",
  "classifier": "POWER",
  "tags": [
    {
      "tag": "source",
      "properties": [
        {"key": "created_via", "value": "api", "value_type": "string"}
      ]
    }
  ]
}

How to search once a datasource is indexed, by identifier, tag property or classifier, is described in Datasource Search.

Creating and Updating Datasources

Append-Only Design Pattern

The Energyworx platform follows an append-only design pattern for resources with built-in versioning, including datasources, tags, and datapoints. This design ensures data integrity, maintains complete audit trails, and supports regulatory compliance requirements.

How Datasource Creation Works

When you create a datasource using the API (e.g., /ewx/v1/datasource/create), the platform handles both creation and updates through the same endpoint:

• New Datasource: If the datasource ID does not exist, a new datasource is created with the provided properties
• Existing Datasource: If the datasource ID already exists, the datasource properties are updated with the new values

This behavior is consistent with the platform's append-only architecture, where resources with built-in versioning support updating properties through creation endpoints rather than having separate update endpoints.

Important Considerations

No Duplicate Prevention: The API does not prevent you from calling the create endpoint with an existing datasource ID. Instead, it will update the existing datasource's properties. This means:

• You will receive a successful response (200 OK) even when updating an existing datasource
• The endpoint name "create" may be misleading, as it also handles updates
• There is no separate dedicated update endpoint for datasources in API v1.5

Best Practices:

• Be aware that calling the create endpoint multiple times with the same ID will update the datasource each time
• If you need to verify whether a datasource exists before creating it, use the get endpoint first
• For production workflows, ensure your integration logic handles this idempotent behavior appropriately
• Consider implementing client-side checks if you need to distinguish between create and update operations

Re-ingestion Workflows: This behavior supports re-ingestion scenarios where datasources need to be recreated or refreshed. The platform allows you to send the same datasource data multiple times without errors, updating properties as needed.

Example

// First call - creates a new datasource
POST /ewx/v1/datasource/create
{
  "id": "METER_001",
  "classifier": "POWER",
  "description": "Main building meter"
}
// Response: 200 OK - Datasource created

// Second call with same ID - updates the existing datasource
POST /ewx/v1/datasource/create
{
  "id": "METER_001",
  "classifier": "POWER",
  "description": "Updated description for main building meter"
}
// Response: 200 OK - Datasource properties updated

Datasource Deletion Policy

Append-Only Platform Design

The Energyworx platform is designed as an append-only system for auditing purposes. This means that datasource deletion is not available through the standard platform interface or API. This design ensures data integrity, maintains complete audit trails, and supports regulatory compliance requirements.

Deletion in Non-Production Environments

For development and testing environments only, Energyworx provides a datasource deletion service on a best-effort basis. If you need to delete datasources from a non-production environment:

1. Submit a request through the Energyworx Service Desk
2. Provide the following information:
   • Environment name (e.g., ewx-dev-001, ewx-staging-002)
   • Namespace ID
   • List of datasource IDs to be deleted

Important Considerations:

• Best-Effort Service: While we make every effort to remove all related data, some references to deleted datasources may persist in:

  • Audit events
  • Search indices
  • Flow execution history
  • Other platform objects that reference the datasources

• Impact on Platform Experience: Lingering references may affect functionality in:

  • Search results
  • Dashboards
  • Reports
  • Historical flow executions

• Production Environments: Datasource deletion is not recommended and typically not permitted in production environments due to the platform's design principles.

Alternative: Deactivating Datasources

Instead of deletion, consider deactivating or marking datasources as inactive:

• Stop ingesting new data for the datasource
• Update datasource tags to indicate inactive status
• Use datasource filters in searches and reports to exclude inactive datasources

This approach maintains the audit trail while reducing clutter in active workflows.

Multiple Datasources View

This article describes how to create a graph with timeseries data from multiple datasources.

1. Go to Search Datasources.
2. Search and select the datasources that you want to view.
3. Click on “Create view from selection” in the upper-right.
4. Select the date range you want to see.
5. Click “View Datasources”, you will be directed to a view with multiple datasources.
6. Scroll down to the chart, and select data to be plotted.

See below a video for this.