Quick Start - SFTP Integration

This guide walks you through adding a new SFTP integration step-by-step.

Prerequisites

Access to GCP project with Secret Manager permissions
Access to the configuration storage bucket (<project-id>-integration-service)
SFTP server credentials (host, username, password, host key fingerprint)
Understanding of the message format you'll be processing

Keep configuration under version control

Keep all configuration files under version control in addition to uploading them to the bucket, so changes are reviewable and reproducible.

The upload examples below use gsutil. If you do not have direct GCP access, the configuration bucket can also be reached over SFTP/S3 — see How to connect to the Integrations Config Bucket using WinSCP.

Step-by-Step Process

Step 1: Store SFTP Credentials in Secret Manager

Navigate to Google Cloud Console → Secret Manager
Click "CREATE SECRET"
Name: <connection_id>_password (e.g., acme_sftp_dev_password). A good connection_id follows the format source-type-stage.
Secret value: Enter the SFTP password. If the password is not yet available, create the secret with a placeholder value and update it later — the connection will fail until the real password is set.
Click "CREATE SECRET"

Example:

Secret name: acme-sftp-password
Secret value: your_actual_password

Step 2: Store SFTP Host Key in Secret Manager

To securely verify the SFTP server's identity, you need to store its SSH host key.

Get the Host Key Fingerprint

Option 1: Using ssh-keyscan (Recommended)

# Get the host key
ssh-keyscan -p 22 -H sftp.example.com

# Example output:
# |1|base64hash...|base64hash... ssh-rsa AAAAB3NzaC1yc2EAAAADAQAB...

Option 2: Manual SFTP connection

sftp -P 22 username@sftp.example.com
# You'll see: "The authenticity of host 'sftp.example.com' can't be established."
# RSA key fingerprint is SHA256:abc123def456...
# Copy the entire key shown

Option 3: From known_hosts file

# If you've connected before, check your known_hosts
cat ~/.ssh/known_hosts | grep sftp.example.com

Get the ed25519 one to add in secret manager, if the server supports it. If not, get the rsa key.

Store the Host Key in Secret Manager

Navigate to Google Cloud Console → Secret Manager
Click "CREATE SECRET"
Name: MUST follow pattern: <connection_id>_trusted_server_key
- Example: acme_sftp_trusted_server_key
- If the host key is not yet available, create the secret with a placeholder value and update it later.
Secret value: Paste the complete host key line from ssh-keyscan output
Click "CREATE SECRET"

Example:

Secret name: acme_sftp_trusted_server_key
Secret value: |1|base64hash...|base64hash... ssh-rsa AAAAB3NzaC1yc2EAAAADAQAB...

⚠️ Important: The secret name MUST match the pattern {connection_id}_trusted_server_key where {connection_id} is the ID you'll use in your connection config (Step 5).

Step 3: Host Key Secret Path

The host key secret path pattern is configured on the platform side in settings_sftp.toml:

[default]
TRUSTED_KEY_SECRET_PATH = "projects/<project-number>/secrets/{id}_trusted_server_key"
MAX_RETRIES = 3
RETRY_DELAY_SECONDS = 2
# Large-file handling (overridable per connection via the connection config)
SFTP_STREAM_THRESHOLD_BYTES = 268435456   # 256 MiB — above this, files are streamed to File Manager
SFTP_MAX_FILE_SIZE_BYTES = 5368709120     # 5 GiB — above this, files are skipped
SFTP_SKIPPED_FOLDER = "skipped"           # folder (under source_folder) skipped files are moved into

The {id} placeholder will be automatically replaced with your connection's id field.

Example: If your connection ID is acme_sftp, the system will look for:

projects/<project-number>/secrets/acme_sftp_trusted_server_key

Step 4: Test SFTP Connection Manually

Before configuring, verify the connection works:

sftp -P 22 username@sftp.example.com
# Enter password when prompted
# List files: ls /outbound
# Exit: exit

Step 5: Create Connection Configuration File

Create file: connections/<connection_id>.json

Template for Basic Authentication:

{
  "id": "your_connection_id",
  "host": "sftp.example.com",
  "port": 22,
  "auth": {
    "type": "basic",
    "username": "sftp_username",
    "pw_secret_key": "projects/<project-id>/secrets/your-connection-id-sftp-password"
  },
  "source_folder": "/outbound",
  "allowed_extensions": ["xml", "txt", "csv"]
}

Example: connections/acme_sftp.json

{
  "id": "acme_sftp",
  "host": "sftp.acme.example.com",
  "port": 22,
  "auth": {
    "type": "basic",
    "username": "energyworx_inbound",
    "pw_secret_key": "projects/<project-id>/secrets/acme-sftp-password"
  },
  "source_folder": "/outbound/dre",
  "allowed_extensions": ["xml"]
}

Template for Certificate Authentication:

{
  "id": "your_connection_id",
  "host": "sftp.example.com",
  "port": 22,
  "auth": {
    "type": "certificate",
    "transport_cert": {
      "name": "transport-cert",
      "data": "-----BEGIN CERTIFICATE-----\nMIIC...\n-----END CERTIFICATE-----",
      "expiration_date": "2027-12-31"
    },
    "signing_cert": {
      "name": "signing-cert",
      "data": "-----BEGIN CERTIFICATE-----\nMIIC...\n-----END CERTIFICATE-----",
      "expiration_date": "2027-12-31"
    }
  },
  "source_folder": "/outbound",
  "allowed_extensions": ["xml"]
}

Field Notes:

id: Must match the connection_id used in schedule config and host key secret name
host: SFTP server hostname or IP address
port: SFTP port (default: 22)
auth.type: Either "basic" or "certificate"
auth.pw_secret_key: Full Secret Manager resource path of the password secret (projects/<project-id>/secrets/<secret-name>) for basic auth
source_folder: Path on SFTP server to retrieve files from (optional)
allowed_extensions: File extensions to process (optional, defaults to all)

Upload to bucket:

gsutil cp connections/acme_sftp.json gs://<project-id>-integration-service/connections/acme_sftp.json

Step 6: Add Schedule Configuration

Edit/Add: schedules/schedule_config.json in the configuration bucket.

Add your schedule to the "schedules" array:

{
  "schedule_types": {
    "every_15_minutes": {
      "description": "Runs every 15 minutes",
      "interval": "15m",
      "timezone": "UTC",
      "type": "interval"
    },
    "every_5_minutes": {
      "description": "Runs every 5 minutes",
      "interval": "5m",
      "timezone": "UTC",
      "type": "interval"
    },
    "every_minute": {
      "description": "Runs every minute",
      "interval": "1m",
      "timezone": "UTC",
      "type": "interval"
    }
  },
  "schedules": [
    {
      "name": "acme_dre_schedule",
      "enabled": true,
      "type": "sftp",
      "connection_id": "acme_sftp",
      "schedule": "every_5_minutes",
      "namespaces": ["enrx_org_004"],
      "description": "Process DRE meter reading files from Acme"
    }
 ]
}

Upload to bucket:

gsutil cp schedules/schedule_config.json gs://<project-id>-integration-service/schedules/schedule_config.json

Step 7: Configure Message Type

Edit: message_config.json

Add your message type:

"DRE": {
  "mmchub_inbound_type": false,
  "sftp_inbound_type": true,
  "xsd_schemas": {
    "folder": "schemas",
    "file": "DRE_MeterData.xsd"
  },
  "message_maps": {},
  "rule_configs": []
},
"IDE": {
    "message_maps": {},
    "rule_configs": [],
    "sftp_inbound_type": true,
    "xsd_schemas": {}
  }

For simple SFTP without validation:

"DRE": {
  "mmchub_inbound_type": false,
  "sftp_inbound_type": true,
  "xsd_schemas": {},
  "message_maps": {},
  "rule_configs": []
}

Upload to bucket:

gsutil cp message_config.json gs://<project-id>-integration-service/message_config.json

Step 8: Create Field Map Configuration

Create a file field_map_configs/<message_type>.json for each message type (e.g. DRE.json and IDE.json).

Template:

{
  "field_maps": {
    "inbound_filename": {
      "function": "concat_hyphen",
      "args": [
        {"function": "return_string", "args": "dre"},
        {"function": "current_date_time", "args": ""}
      ]
    },
    "message_id": {
      "function": "concat_hyphen",
      "args": [
        {"function": "return_string", "args": "dre"},
        {"function": "current_date_time", "args": ""}
      ]
    },
    "domain": {
      "function": "return_string",
      "args": "domain"
    },
    "allocation_date": {
      "function": "current_date_time",
      "args": ""
    },
    "correlation_id":  {
      "function": "return_string",
      "args": ""
    },
    "sender_id": {
      "function": "return_string",
      "args": "Acme"
    },
    "receiver_id": {
      "function": "return_string",
      "args": "Energyworx"
    },
    "creation_timestamp": {
      "function": "current_date_time",
      "args": ""
    },
    "process_type": {
      "function": "return_string",
      "args": "dre"
    },
    "process_date": {
      "function": "current_date",
      "args": ""
    }
  }
}

Upload to bucket:

gsutil cp field_map_configs/DRE.json gs://<project-id>-integration-service/field_map_configs/DRE.json
gsutil cp field_map_configs/IDE.json gs://<project-id>-integration-service/field_map_configs/IDE.json

Step 9: Create File Tags Configuration (optional)

Create a file file_tags/<message_type>.json for each message type that needs custom file tags. Skip this step if the default MessageType and MessageStatus tags are sufficient.

Template:

{
    "Source": "sender_id",
    "MessageID": "message_id",
    "ProcessType": "process_type",
    "CreationTimestamp": "creation_timestamp",
    "CreationDate": "process_date"
}

Upload to bucket:

gsutil cp file_tags/DRE.json gs://<project-id>-integration-service/file_tags/DRE.json
gsutil cp file_tags/IDE.json gs://<project-id>-integration-service/file_tags/IDE.json

Step 10: Create response configuration files

Create a file response_configs/<message_type>.json for each message type (see Message Config for the structure).

Upload to bucket:

gsutil cp response_configs/DRE.json gs://<project-id>-integration-service/response_configs/DRE.json
gsutil cp response_configs/IDE.json gs://<project-id>-integration-service/response_configs/IDE.json

Step 11: (Optional) Add XSD Schema for Validation

If you want to validate XML structure:

Create file: schemas/DRE_MeterData.xsd
Add your XSD schema content
Upload to bucket:

gsutil cp schemas/DRE_MeterData.xsd gs://<project-id>-integration-service/schemas/DRE_MeterData.xsd

Update message_config.json to reference the xsd schema (as shown in Step 7)

Step 12: Verify Configuration Files Are in Bucket

Check all files are uploaded:

gsutil ls -r gs://<project-id>-integration-service/

# Expected output:
# gs://<project-id>-integration-service/connections/acme_sftp.json
# gs://<project-id>-integration-service/field_map_configs/DRE.json
# gs://<project-id>-integration-service/file_tags/DRE.json
# gs://<project-id>-integration-service/message_config.json
# gs://<project-id>-integration-service/schedules/schedule_config.json

Step 13: Test with Sample File

Before enabling the scheduler, test manually:

Place a test file on the SFTP server in the source_folder
Trigger the inbound scheduler manually (if possible) or wait for next scheduled run
Check logs for processing:

# View scheduler logs
gcloud logging read "resource.type=k8s_container AND labels.app=inbound-scheduler" --limit 50 --format json

# View processor logs
gcloud logging read "resource.type=k8s_container AND labels.app=inbound-processor" --limit 50 --format json

Verify file appears in filemanager
Verify file moved to archive folder on SFTP server

Step 14: Monitor First Production Run

Check BigQuery audit events
Check for errors in logs
Verify files are being archived correctly
Confirm file tags are correct in filemanager

Common Issues and Solutions

Issue: Files Not Being Picked Up

Symptoms: Scheduler runs but no files are processed

Checks:

Verify filename matches pattern (contains message type)
Check file extension is in allowed_extensions
Verify source_folder path is correct
Check SFTP credentials are valid
Ensure schedule is enabled ("enabled": true)

Debug:

# Check scheduler logs for your connection
gcloud logging read "resource.type=k8s_container AND labels.app=inbound-scheduler AND jsonPayload.connection_name='acme_dre_schedule'" --limit 10

Issue: Authentication Failed (Error SFTP-000)

Symptoms: "Unable to connect to SFTP server" in logs

Checks:

Verify host and port are correct
Test credentials manually with SFTP client
Check password in Secret Manager is correct
Verify network connectivity from GKE to SFTP server
Check firewall rules allow outbound connections

Issue: Files Not Being Archived

Symptoms: Files processed but remain in source folder

Checks:

Verify archive_folder exists on SFTP server
Check SFTP user has write permissions to archive_folder
Review processor logs for archive errors

Issue: Duplicate Processing

Symptoms: Same file processed multiple times

Checks:

Verify Redis is running and accessible
Check TTL is set correctly (3600 seconds default)
Ensure connection_id is unique and consistent

Issue: Validation Failures

Symptoms: Files retrieved but validation errors in audit

Checks:

Verify XSD schema is correct and uploaded
Test XML against XSD locally
Check field mappings are correct
Review validation rule configurations

Validation Checklist

Before going to production:

File Structure Summary

For each SFTP integration, you need:

gs://<project-id>-integration-service/
├── connections/
│   └── <connection_id>.json          ← SFTP connection details
├── field_map_configs/
│   └── <message_type>.json           ← Field extraction/generation rules
├── file_tags/
│   └── <message_type>.json           ← File tags for filemanager
├── schemas/                           ← (Optional) XSD schemas
│   └── <message_type>.xsd
├── rules/                             ← (Optional) Validation rules
│   └── <message_type>_<process>.json
├── message_config.json                ← Message type definitions
└── schedules/
    └── schedule_config.json           ← Schedule and routing configuration

Next Steps

After successful setup:

Monitor the integration for 24-48 hours
Adjust schedule frequency if needed
Fine-tune validation rules based on real data
Set up alerting for failures
Document any organization-specific requirements
Train support team on troubleshooting

Getting Help

Check logs: Scheduler and Processor pods
Review audit events in BigQuery
Test SFTP connection manually
Verify configuration files in bucket

Prerequisites​

Step-by-Step Process​

Step 1: Store SFTP Credentials in Secret Manager​

Step 2: Store SFTP Host Key in Secret Manager​

Get the Host Key Fingerprint​

Store the Host Key in Secret Manager​

Step 3: Host Key Secret Path​

Step 4: Test SFTP Connection Manually​

Step 5: Create Connection Configuration File​

Step 6: Add Schedule Configuration​

Step 7: Configure Message Type​

Step 8: Create Field Map Configuration​

Step 9: Create File Tags Configuration (optional)​

Step 10: Create response configuration files​

Step 11: (Optional) Add XSD Schema for Validation​

Step 12: Verify Configuration Files Are in Bucket​

Step 13: Test with Sample File​

Step 14: Monitor First Production Run​

Common Issues and Solutions​

Issue: Files Not Being Picked Up​

Issue: Authentication Failed (Error SFTP-000)​

Issue: Files Not Being Archived​

Issue: Duplicate Processing​

Issue: Validation Failures​

Validation Checklist​

File Structure Summary​

Next Steps​

Getting Help​

Prerequisites

Step-by-Step Process

Step 1: Store SFTP Credentials in Secret Manager

Step 2: Store SFTP Host Key in Secret Manager

Get the Host Key Fingerprint

Store the Host Key in Secret Manager

Step 3: Host Key Secret Path

Step 4: Test SFTP Connection Manually

Step 5: Create Connection Configuration File

Step 6: Add Schedule Configuration

Step 7: Configure Message Type

Step 8: Create Field Map Configuration

Step 9: Create File Tags Configuration (optional)

Step 10: Create response configuration files

Step 11: (Optional) Add XSD Schema for Validation

Step 12: Verify Configuration Files Are in Bucket

Step 13: Test with Sample File

Step 14: Monitor First Production Run

Common Issues and Solutions

Issue: Files Not Being Picked Up

Issue: Authentication Failed (Error SFTP-000)

Issue: Files Not Being Archived

Issue: Duplicate Processing

Issue: Validation Failures

Validation Checklist

File Structure Summary

Next Steps

Getting Help