SFTPGo (heatwave-sftp) — the PBX-facing SFTP gateway, R2-backed

The single SFTP endpoint the Switchvox PBX (144.202.57.170) talks to. It runs
as a Kamal accessory on the Dallas bare-metal host and stores everything it
receives directly in Cloudflare R2 — there is no host-local recordings/backup
mount, so the data survives any app/DB host failover.

Image: drakkan/sftpgo:v2.6.6 (open-source
edition). Accessory definition: config/deploy.yml
→ accessories.sftp. User provisioning: config/sftpgo/bootstrap_call_recordings.sh
(also see config/sftpgo/README.md for the operator runbook).

What it provides

Capability	SFTPGo user	Lands in	Consumer
1. Call-recording ingestion	pbx	R2 heatwave-call-recordings-production, prefix WarmlyYours/	CallRecordImporterWorker reads R2 → transcription/analysis
2. PBX system backups	pbx-backup	R2 heatwave-pbx-backups-production (ENAM), bucket root	DR archive (Switchvox rotates by retention count)
3. WebAdmin / WebClient UI	—	tailnet :8080	Humans: manage users, browse the R2 buckets, watch live connections
4. SMTP notifier	—	SendGrid relay	SFTPGo email (password resets + Event Manager notifications)

Each Switchvox feature points at a distinct SFTPGo user, and each user maps to
its own R2 bucket — so a leaked credential for one can't touch the other, and
backup retention/lifecycle is independent of recordings.

Topology

 Switchvox PBX (144.202.57.170)                 Cloudflare R2 (account 79b7f58…)
        │                                        ├─ heatwave-call-recordings-production/WarmlyYours/
        │  SFTP (Use SFTP = YES)                 └─ heatwave-pbx-backups-production/   (ENAM, off-Latitude)
        ▼                                                  ▲              ▲
 67.213.118.15 : 2222  ──DOCKER-USER firewall──►  heatwave-sftp accessory │ (pbx user, s3 home)
   (public; locked to the PBX IP)                 dal-latitude-heatwave-01 │
                                                  container :2022 (SFTP)   │ (pbx-backup user, s3 home)
 100.123.47.52 : 8080  ─────tailnet only────────► container :8080 (WebAdmin/WebClient UI)
   (no public exposure)

 Rails: CallRecordImporterWorker ──reads──► heatwave-call-recordings-production/WarmlyYours/*.wav

Property	Value
Host	dal-latitude-heatwave-01 (Dallas Latitude; also the PG18 primary box)
Public IP	67.213.118.15 — SFTP only, port 2222
Tailnet IP	100.123.47.52 — WebAdmin UI only, port 8080
SFTP port	host 2222 → container 2022 (SFTPGO_SFTPD__BINDINGS__0__PORT=2022)
Firewall	host-level DOCKER-USER rule allows 2222 only from 144.202.57.170 (the PBX). Scoped with --ctorigdstport 2222 in infra/terraform/latitude/cloud-init.yaml.tftpl. The PBX-side backup feature originates from the same IP, so no rule change was needed to add it.
R2 endpoint	https://79b7f58cf035093b5ad11747df30369a.r2.cloudflarestorage.com (force_path_style, region: auto)

Why R2-backed (the re-architecture)

The recordings server was historically plain FTP/SFTP writing to a host-local
/data/callrecords mount (ProFTPD on a Vultr box, later an atmoz/sftp
accessory). Both tied the pipeline to one host's disk. Re-architected 2026-06-10
to SFTPGo with an S3/R2 filesystem backend: the PBX still speaks SFTP, but
SFTPGo writes objects straight to R2, so the importer (and any future app/DB
failover) depends only on R2, never on a shared host mount.

The Switchvox "create directory" false-positive

Switchvox verifies its configured FTP/SFTP path by trying to create it as a
directory before each run. Against object storage (R2 has only virtual
directories) that check false-fails — Could not create the directory (WarmlyYours) — on every cycle, even though the uploads themselves succeed. The
fix, applied to every Switchvox feature: point its path at the login root /
(which always exists) and let SFTPGo place the objects:

• The pbx user's s3config.key_prefix: "WarmlyYours/" maps its root onto the
  WarmlyYours/ prefix the importer already reads.
• The pbx-backup user has no prefix, so backups land at its bucket root.

Gotcha: Switchvox caches the path in its verification step. After changing a
path, the alarm keeps firing (still naming the old path) until the PBX itself is
rebooted. The recordings call-recording offload also throws a separate,
path-independent "backup directory verification" alarm that is a Switchvox-internal
false-positive on object storage and is muted at the PBX (SV_CALL_RECORDING).

Users & buckets

Provisioned at container start by bootstrap_call_recordings.sh,
which writes a loaddata JSON and runs sftpgo serve --loaddata-from … --loaddata-mode 0
(mode 0 = add new / update existing; restores both users on every reboot).

User	Bucket	Prefix	Credentials
pbx	heatwave-call-recordings-production	WarmlyYours/	container default chain (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY from op://IT/R2-call-recordings-production)
pbx-backup	heatwave-pbx-backups-production	— (root)	explicit per-user s3config.access_key/access_secret — a dedicated bucket-scoped token (op://IT/R2-pbx-backups-production), not the recordings creds

The pbx-backup secret is given to SFTPGo as {"status":"Plain","payload":"…"};
SFTPGo's builtin KMS encrypts it (AES-256-GCM) on load — no master key needed.

Switchvox configuration

Both features live in the Switchvox admin. Use the same host:port for each;
only the user/password/path differ. Use SFTP = YES (the server is SFTP-only on
2222 — it does not serve plain FTP).

Call recording offload (Setup → Call Recording):

Field	Value
Host / port	67.213.118.15:2222
Username	pbx
Password	op://IT/R2-call-recordings-production → pbx_password
Path	/

System backups (Setup → Set Up Automatic Backups):

Field	Value
Use SFTP	YES
Host / port	67.213.118.15:2222
Username	pbx-backup
Password	op://IT/R2-pbx-backups-production → pbx_backup_password
Path	/
Number of backups to keep	operator's choice — Switchvox rotates via SFTP DELETE (the user has * perms)

WebAdmin / WebClient UI

Published on the tailnet IP only (options.publish: 100.123.47.52:8080:8080) —
never on the public interface. From any tailnet-connected machine:

• Admin console: http://100.123.47.52:8080/web/admin — login admin /
  op://IT/SFTPGo (the dedicated item also holds the URL).
  • ⚠️ The bare :8080/ redirects to /web/client/login (the end-user portal,
    which only knows the SFTP users); logging in there as admin returns
    "user not found". Always use /web/admin.
• The admin password is seeded only at first data-provider init
  (CREATE_DEFAULT_ADMIN); rotating SFTPGO_DEFAULT_ADMIN_PASSWORD later does
  not update an already-created admin (reset via a loaddata admins block).

SMTP notifier

SFTPGo sends mail through the SendGrid SMTP relay (op://IT/SendGrid), so it
can email password-reset codes and — via Event Manager rules — operational
notifications.

SFTPGO_SMTP__*	Value
HOST / PORT	smtp.sendgrid.net / 587
ENCRYPTION	2 (STARTTLS)
AUTH_TYPE	0 (PLAIN)
USER	apikey (literal)
PASSWORD	the SendGrid send-only API key (op://IT/SendGrid → "Send-only API key…"), read by a direct op read in .kamal/secrets-common
FROM	WarmlyYours SFTPGo <systems@warmlyyours.com>

The SMTP config is the transport only. To actually emit alerts, add an
Event Manager rule (WebAdmin → Events, or a loaddata event_rules block).
Note SFTPGo upload-event rules fire only when the PBX connects; "the backup
silently stopped running" is better caught by an external daily freshness check.

Credentials (1Password, IT vault)

Item	Used for
R2-call-recordings-production	pbx SFTP password, recordings-bucket R2 token (AWS_*)
R2-pbx-backups-production	pbx-backup SFTP password + dedicated backups-bucket R2 token; minted/rotated by script/setup_r2_pbx_backups_token.sh
SFTPGo	WebAdmin admin login + URL
SendGrid (systems@warmlyyours.com)	SMTP relay send-only API key

All are resolved in .kamal/secrets-common and
injected via the accessory's env.secret (Kamal redacts them in logs).

Operations

Apply a config change (bootstrap, env, or secrets) — kamal deploy does not
recreate accessories, so reboot it explicitly. The -d production host list
includes an unreachable host, so always scope to the SFTP host:

mise exec -- bundle exec kamal accessory reboot sftp -d production --hosts 100.123.47.52

A reboot re-runs the bootstrap (restores both users from the loaddata JSON);
expect restoring existing user: "pbx" / "pbx-backup" with error: <nil> and
listeners on :2022 + :8080.

Add / change a user — edit the loaddata JSON in
bootstrap_call_recordings.sh
and reboot. Validate JSON shape in a throwaway container first
(docker run --rm … drakkan/sftpgo:v2.6.6 -lc bootstrap…) before touching prod.

Rotate the backups R2 token — re-run script/setup_r2_pbx_backups_token.sh
(preserves the existing SFTP password) and reboot.

Monitoring

• Importer: CallRecordImporterWorker (hourly 6am–7pm CT) in the Sidekiq
  dashboard; AppSignal for SFTP/R2 errors.
• Live SFTP sessions & file listings: the WebAdmin UI.
• Backups landing: list heatwave-pbx-backups-production, or watch the
  pbx-backup user's activity in the UI. First scheduled Switchvox run is nightly.

Import side (recordings)

The recordings half of the pipeline (the Rails consumer) is documented in
Call Recording System. In short:
config.x.call_record_importer.storage = :r2 (prod/staging) routes
CallRecordImporterWorker → CallRecordSwitchvoxImporterCloudStorage →
CallRecordSwitchvoxObjectStore, which reads *.wav under WarmlyYours/ using the
call_recordings.object_store.* encrypted credentials (the bucket name must contain
the environment string). Only .wav is imported; stray non-.wav objects are
ignored.

Real-time import hook (upload action)

By default the importer polls hourly. SFTPGo's upload action hook makes the
import event-driven: the instant the PBX finishes writing a recording, SFTPGo
POSTs an FsActionNotification to the Rails app, which enqueues the import
immediately — recordings appear in the CRM within seconds instead of waiting for
the next poll.

PBX ──SFTP──▶ SFTPGo ──upload action hook (POST JSON)──▶
   api.warmlyyours.com/webhooks/v1/sftpgo?token=…
      └▶ Webhooks::V1::SftpgoController
           └▶ WebhookLog.ingest!(provider: sftpgo, external_id: <wav R2 key>)
                └▶ WebhookProcessorWorker → WebhookProcessors::SftpgoProcessor
                     └▶ CallRecordImporterWorker.perform_async(<wav R2 key>)

Design points:

• Trigger on the .xml sidecar, not the .wav. Switchvox writes a paired
  <name>.wav + <name>.xml; keying off the metadata file guarantees the
  importer has both halves (a fractionally-late .wav is covered by the worker's
  retry). The controller hard-filters to the pbx user's .xml uploads and ACKs
  everything else (the .wav event, pbx-backup archives, archive moves).
• The hourly poll stays as the backstop. Both paths call
  CallRecordImporterWorker.perform_async(<wav R2 key>) with the same key, so
  the worker's :until_executed lock + the importer's first_or_initialize +
  the post-import move to processed/ make a webhook+poll race a guaranteed
  no-op. Nothing is imported twice; nothing the hook drops is lost.
• Auth is a shared-secret token query param (secure_compare), failing
  closed in production when unset.

Activation (one-time wiring — currently DORMANT until set)

The Rails endpoint ships ready; SFTPGo is not yet pointed at it. To turn it on:

1. Mint a token: openssl rand -hex 24.

2. Make the app trust it — set SFTPGO_WEBHOOK_TOKEN in the web role's
   env.secret (via .kamal/secrets-common ← 1Password), or add
   sftpgo: { webhook_token: … } to Rails credentials. The controller reads
   ENV['SFTPGO_WEBHOOK_TOKEN'] first, then Heatwave::Configuration.fetch.

3. Make SFTPGo call it — in the sftp accessory (config/deploy.yml):

   env:
     clear:
       SFTPGO_COMMON__ACTIONS__EXECUTE_ON: upload
     secret:
       - SFTPGO_COMMON__ACTIONS__HOOK   # https://api.warmlyyours.com/webhooks/v1/sftpgo?token=<TOKEN>
   

   Store the full URL (token embedded) in 1Password → .kamal/secrets-common.
   The hook fires for every upload/user; the controller does the filtering, so
   no per-user SFTPGo config is needed.

4. kamal deploy (reboots the web role + the sftp accessory).

This uses SFTPGo's simple global Custom Actions hook (common.actions),
configured entirely from env. SFTPGo also offers the newer rule-based Event
Manager (per-user/path filters, configured via the data provider) — not needed
here because the controller already does the filtering, and the env-only hook
keeps all config in deploy.yml rather than in provider state. If we later want
SFTPGo-side filtering, the Event Manager is the migration path.

A failed hook is logged by SFTPGo and never blocks the upload — the poll still
imports it — so activation is low-risk and reversible (drop the two env keys).

Related documentation

• Call Recording System — the Rails feature (import, transcription, UI).
• Switchvox Webhooks — the other Switchvox integration.
• config/sftpgo/README.md — operator quick-reference next to the bootstrap.