Infrastructure & Monitoring Inventory

The single-page answer to “what runs where, on which IP and port, what protects it, and what URL do I use to reach it.” This is hand-maintained reference — the source of truth is the config it summarizes (config/deploy.yml, config/deploy.staging.yml, config/netdata/, config/haproxy/, infra/terraform/). Keep it in sync when those change.

Last verified: 2026-08-01 (revision e34f830d48; PG18, prod-on-Kamal, Dallas-primary / Chicago-standby, three isolated Sidekiq roles).

Just want the link to a service? The Service Directory & Access page is the one-screen launchpad — every URL, what it does, and where the login lives. This page is the deeper what-runs-where reference behind it.

Fleet — physical hosts (where)

Two Latitude.sh bare-metal boxes (f4.metal.medium: AMD EPYC 4564P 16c/32t, 192 GB RAM, 2×480 GB NVMe RAID1 + 2×2 TB NVMe, 2×10 GbE). The Kamal config addresses them exclusively by tailnet IP; the public IPs are physical-host facts (used for SSH-over-tailnet only — there is no public SSH).

Host	DC	Public IP	Tailnet IP	Role today
`dal-latitude-heatwave-01`	Dallas	`67.213.118.15`	`100.123.47.52`	Prod app + DB primary, and the entire staging stack (co-located)
`chi-latitude-heatwave-02`	Chicago	`186.233.186.45`	`100.68.157.49`	Prod DB standby + Databasus PITR + four-guest Incus CI-runner pool + Dozzle log hub (no prod app today)

At a planned Chicago cutover (“W3”) the app role flips to chi-02 via pg_promote + HAProxy reroute — see Failover & maintenance tooling below. Today, all production web/Sidekiq traffic is served from Dallas.

                 Internet
                    │  (HTTPS)
            ┌───────▼────────┐
            │   Cloudflare    │  edge TLS, www-edge Worker, WAF
            └───────┬────────┘
                    │  outbound-only QUIC tunnel (cloudflared, host systemd)
        ┌───────────▼───────────── dal-01 (Dallas, 100.123.47.52) ──────────────┐
        │  kamal-proxy :80 ─▶ web (Thruster → Puma)                             │
        │  app workers: sidekiq latency · sidekiq_heavy · sidekiq_edi            │
        │        │                                                              │
        │        ▼  DATABASE_HOST=heatwave-haproxy:6433                          │
        │  HAProxy :6433 (TCP write-VIP) ─▶ pgbouncer :6432 ─▶ Postgres :5432    │
        │        ▲ httpchk pg-health :8008                    (PG18 PRIMARY)     │
        │  Valkey ×3 (cache / sessions / queue) · Playwright · SFTPGo · Netdata  │
        └───────────────────────────────┬───────────────────────────────────────┘
                                         │ streaming replication (slot chicago_standby)
        ┌────────────────────────────────▼──── chi-02 (Chicago, 100.68.157.49) ──┐
        │  Postgres :5432 (PG18 STANDBY) ◀ pgbouncer :6432 (RO VIP heatwave-db-ro)│
        │  pg-health :8008 · Netdata · Databasus controller (PITR → R2) + UI :4005│
        └─────────────────────────────────────────────────────────────────────────┘

Administrative services — URLs to use (how to access)

Get on the tailnet first (tailscale up; the boxes only accept SSH and admin UIs from 100.64.0.0/10). Admin logins live in 1Password vault IT.

Service	URL	Reach / gating	What it shows
Netdata — prod (Dallas)	`http://100.123.47.52:19999`	tailnet only	Per-second host + every container + Postgres (primary) + Valkey ×3 + pgbouncer + HAProxy + systemd units
Netdata — standby (Chicago)	`http://100.68.157.49:19999`	tailnet only	Standby host + Postgres replica (recovery state / replication lag) + RO pgbouncer
Netdata — staging	`ssh -L 19999:localhost:19999 deploy@100.123.47.52` → `http://localhost:19999`	localhost (SSH-forward)	Staging stack (prod owns the tailnet `:19999` on the shared box)
Dozzle (log hub)	`http://100.68.157.49:9999`	tailnet only	Read-only live container-log tail across chi-02 + the dal-01 agent. Ephemeral CI runner diagnostics are archived on chi-02 by their reset watcher. Login `op://IT/Dozzle-<you>` (per-dev)
Incus web UI	`https://100.68.157.49:8443/ui/`	tailnet only	Manage the chi-02 VMs/containers, including `ci-runner-a` through `ci-runner-d`. Auth = TLS client cert (`bin/incus-ui-cert`)
PgHero	`https://crm.warmlyyours.com/pghero`	Rails admin login (`crm.*` host, `is_admin?`)	Slow queries (pg_stat_statements), table/index bloat, live queries, vacuum, index suggestions (hypopg)
Sidekiq Web UI	`https://crm.warmlyyours.com/sidekiq`	Rails admin login	Background-job queues, retries, scheduled set
Rails Event Store browser	`https://crm.warmlyyours.com/res`	Rails admin login	The RES domain-event stream
HAProxy stats	`http://100.123.47.52:8404/`	tailnet only	DB write-VIP backend health (which node is “up”)
SFTPGo admin	`http://100.123.47.52:8080`	tailnet only	SFTP users/sessions (call-records + PBX backups → R2)
Databasus (PITR controller)	`http://100.68.157.49:4005`	tailnet only	Configure/monitor backup sources; trigger PITR restores. Login `op://IT/Databasus (Postgres Backup)`
Mailpit — staging	`http://100.123.47.52:8025`	tailnet only	Captured staging outbound mail
Mailpit — dev	`http://localhost:8025`	local docker-compose	Captured dev outbound mail
AppSignal	`https://appsignal.com/warmlyyours`	SaaS login	APM, exceptions, traces (apps `Heatwave/production` + `Heatwave/staging`)
Latitude.sh	`https://www.latitude.sh/dashboard`	SaaS login	Bare-metal host provider — the two `f4` boxes, edge firewall, remote console/reinstall, billing
Netdata Cloud	`https://app.netdata.cloud/spaces/warmlyyours/rooms/all-nodes/home`	Google SSO	Single pane over both Netdata agents (outbound ACLK) + centralized alerting; data stays local on the agents
Cloudflare	`https://dash.cloudflare.com/79b7f58cf035093b5ad11747df30369a/warmlyyours.com`	SaaS login	DNS, `cloudflared` tunnel, WAF/cache rules, Workers, R2, Access. Zone rulesets are Terraform-managed (dashboard read-only for those)
HCP Terraform	`https://app.terraform.io/app/warmlyyours`	SaaS login	IaC runs/state (Latitude, host-config, Tailscale, Cloudflare zones)

Service topology (what runs where) — production

All ports below are bound to the tailnet IP unless noted. App→service wiring goes over the internal kamal Docker network by DNS name, not these host ports (host ports are for operators). Image tags here are indicative — config/deploy.yml is authoritative for exact tags/digests (e.g. the haproxy @sha256 pin).

Dallas — `dal-latitude-heatwave-01` (`100.123.47.52`)

Service (container)	Port (host→ctr)	Image	Purpose
`kamal-proxy`	(CF tunnel → `:80`)	kamal-proxy	Rolling-deploy reverse proxy; health `/up`; routes `crm/www/api/scan/mcp.warmlyyours.com`
web / sidekiq / sidekiq_heavy / sidekiq_edi	none published	`ghcr.io/warmlyyours/heatwave`	Rails (Puma via Thruster :80→:3000); isolated latency, heavy, and EDI Sidekiq roles
`heatwave-postgres` (primary)	`5432`	`ghcr.io/warmlyyours/heatwave-postgres:18-noble`	PG18.4 RW primary; data on ZFS `tank/prod-replica`
`heatwave-pgbouncer` (RW)	`6432`	`…/heatwave-pgbouncer:1.25.2`	Session-mode pooler in front of the primary
`heatwave-haproxy` (write VIP)	`6433` (+ `8404` stats)	`haproxy:3.0-alpine`	TCP failover router — the app’s `DATABASE_HOST`; httpchk on each node’s pg-health keeps only the live primary “up”
`heatwave-pg-health`	`8008`	`…/heatwave-pg-health:v3`	HTTP leader probe (`200 primary` / `503 standby`) for HAProxy
`heatwave-valkey-cache`	none (kamal-net)	`valkey/valkey:9.1`	Logical DBs 1,2,4,5 · `allkeys-lru` · no persistence
`heatwave-valkey-sessions`	none	`valkey/valkey:9.1`	Logical DB 0 · `noeviction` · no persistence
`heatwave-valkey-queue`	none	`valkey/valkey:9.1`	Logical DB 3 (Sidekiq) · `noeviction` + AOF/RDB (durable)
`heatwave-sftp` (SFTPGo)	`2222`→2022 (public, PBX-only); `8080` UI (tailnet)	`drakkan/sftpgo:v2.6.6`	Switchvox call-records + PBX-backup drop → Cloudflare R2
`heatwave-playwright`	none (`:3000`)	`mcr.microsoft.com/playwright:v1.60.0-noble`	Headless browser for server-side PDF/email/upload flows
`heatwave-netdata`	`19999`	`netdata/netdata:v2.10.3`	Per-second observability (host + all of the above)

Production application capacity

The 2026-07-31 rollout split the former 49-thread Sidekiq process into three failure domains. These are application roles on the same Dallas host and deploy the same revision as web; they are not accessories.

Role	Runtime capacity	DB pool per process	Memory policy
`web`	4 Puma processes × 5 request threads	7 each (28 total)	no role-specific cgroup cap
`sidekiq`	29 latency/operational threads; sole scheduler	38	warn 6 GiB; recycle after GC at 8 GiB; hard cap 12 GiB
`sidekiq_heavy`	19 aggregate/API/media threads	20	warn 6 GiB; recycle after GC at 8 GiB; hard cap 12 GiB
`sidekiq_edi`	4 product-feed threads	5	warn 12 GiB; recycle after GC at 18 GiB; hard cap 24 GiB

Production therefore has 52 Sidekiq threads and 91 potential application connections per database (web 28 + workers 38 + 20 + 5), below PgBouncer’s 92-connection per-database ceiling. Staging uses an EDI pool of 4 and totals exactly 90. Queue ownership, drain grace, monitoring, and the tuning rule live in Sidekiq operations; connection arithmetic lives in PgBouncer connection pooling.

Chicago — `chi-latitude-heatwave-02` (`100.68.157.49`)

Service (container)	Port	Image	Purpose
`heatwave-postgres-replica` (standby)	`5432`	`…/heatwave-postgres:18-noble`	PG18 streaming standby (slot `chicago_standby`); read-offload target; promoted at flip
`heatwave-pgbouncer-replica` (RO)	`6432`	`…/heatwave-pgbouncer:1.25.2`	RO pooler behind the `heatwave-db-ro` tailnet VIP
`heatwave-pg-health-replica`	`8008`	`…/heatwave-pg-health:v3`	Standby recovery-state probe for HAProxy
`heatwave-netdata-replica`	`19999`	`netdata/netdata:v2.10.3`	Standby host + replica lag
Databasus controller	`4005` (UI)	`databasus/databasus:v3.50.0` (official)	Agentless PITR backups (controller-driven `pg_basebackup` + `pg_receivewal`) → R2
`heatwave-dozzle` (log hub)	`9999`→8080	`amir20/dozzle:v10.6.10`	Read-only container-log UI; aggregates chi-02 locally + dal-01 (`7007` agent). Kamal accessory (`kamal accessory reboot dozzle`)
Incus daemon + web UI	`8443` (UI)	host pkg (Ubuntu archive `incusd` + Zabbly `incus-ui-canonical`)	Runs the four `ci-runner-` KVM guests on ZFS `tank/incus`; not* Kamal-managed. TLS-client-cert auth

dal-01 also runs heatwave-dozzle-agent (amir20/dozzle, host port 7007, cmd: agent) — the log source the chi-02 hub dials.

Container images come from GHCR (ghcr.io/warmlyyours/…); registry auth is per-developer via the gh CLI (no shared PAT). Secrets resolve through Kamal’s 1Password adapter (vault IT) — only names are referenced in config, never values.

Port-exposure map (the crux)

Port	Bind	Service	Host	Notes
80/443	CF tunnel only	web (kamal-proxy)	Dallas	No inbound web port open; DOCKER-USER drops public 80/443
2222	public → PBX IP only	SFTPGo SSH	Dallas	Firewalled to `144.202.57.170` (Switchvox) via edge fw + DOCKER-USER `--ctorigdstport`
22	tailnet	SSH	both	Latitude edge fw + UFW allow only `100.64.0.0/10`
5432	tailnet	Postgres	both	primary (Dallas) / standby (Chicago)
6432	tailnet	pgbouncer	both	RW (Dallas) / RO (Chicago)
6433	tailnet	HAProxy write-VIP	Dallas	app DB target
8008	tailnet	pg-health	both	HAProxy httpchk
8404	tailnet	HAProxy stats / `/metrics`	Dallas	stats UI + netdata scrape
8080	tailnet	SFTPGo admin UI	Dallas
19999	tailnet	Netdata	both
4005	tailnet	Databasus UI	Chicago
9999	tailnet	Dozzle log hub	Chicago	Aggregates the 7007 agents
8443	tailnet	Incus web UI	Chicago	TLS client cert; manages the CI-runner guest pool
7007	tailnet	Dozzle agent	Dallas	Only the chi-02 hub dials it
8025	tailnet	Mailpit UI (staging)	Dallas	SMTP 1025 stays kamal-net-internal
6379 / 3000	kamal-net only	Valkey ×3 / Playwright	Dallas	not host-published

Data tier (DB read/write path)

Write path: app DATABASE_HOST=heatwave-haproxy, DATABASE_PORT=6433 → HAProxy (TCP passthrough) → the live primary’s pgbouncer :6432 (session-mode — the app uses session advisory locks) → Postgres :5432. Both heatwave and heatwave_versions share one FDW-linked cluster.
Routing decision: HAProxy httpchks each node’s pg-health :8008; only the node answering 200 "primary" is “up”, so a pg_promote re-routes the write path with no app redeploy.
Read offload: the heatwave-db-ro Tailscale VIP follows the standby’s pgbouncer for read-only consumers.
Cache/queue: RedisConfig (config/initializers/100_redis_config.rb) routes by logical DB to heatwave-valkey-{cache,sessions,queue}:6379 — see Valkey 3-Flavor Split and DB Tier HA Architecture.

Failover & maintenance tooling

bin/db failover <env> {flip-db,rebuild-standby,topology} — promote the standby + reroute the write VIP (cross-DC), rebuild a wiped node from a fresh basebackup, or print the current topology. Snapshots the demoted dataset (zfs snapshot) before wiping, as a rollback net.
bin/db maintenance {up,down} <env> — full maintenance window (proxy 503 → stop web+sidekiq → on prod also stop the Chicago databasus controller container), and the reverse. See the PG18 Failover Runbook and HAProxy Routing Layer.

Backups & disaster recovery

Databasus PITR (backup-of-record) — the agentless databasus controller container on Chicago streams an encrypted physical base + WAL to Cloudflare R2 bucket heatwave-postgres-backups-production (ENAM region, off-Latitude for DR). Restore to any chosen second via the controller UI (:4005) or bin/db restore’s physical/PITR option (wraps config/databasus/databasus-recovery.sh). AES-256-GCM key at /data/databasus-data/secret.key.
Databasus is itself disaster-recoverable (config-as-code) — the controller keeps its entire config (admin / workspace / R2 storage / sources / schedules / restore user) only in an embedded Postgres metadata DB on the host, so a host wipe loses it (it did, 2026-06-15) and only secret.key is in 1Password. script/setup_databasus.sh rebuilds the whole thing idempotently in one command; a nightly encrypted metadata snapshot (databasus-metadata-backup.timer → R2 databasus-metadata/) is the turnkey alternative. See config/databasus/README.md.
heatwave_versions archive — bin/db versions ships completed annual partitions (>5 yr) to R2 heatwave-versions-archive-production (cold).
bin/db restore — pulls a recent logical pg_dump from Databasus→R2 to seed a dev database (this is the dev path, not DR). See the DR restore runbook and Databasus PITR.

Cloudflare R2 buckets (all off-Latitude; bucket-scoped S3 tokens in 1Password vault IT)

Bucket	Region	Purpose	Producer
`heatwave-postgres-backups-production`	ENAM	Databasus PITR (physical base + WAL). `databasus-metadata/` prefix = the encrypted controller-config snapshots	Databasus controller + `backup-metadata.sh`
`heatwave-versions-archive-production`	ENAM	Cold annual `heatwave_versions` partitions (>5 yr)	`bin/db versions`
`heatwave-frontend-assets-production`	ENAM	Content-hashed webpack assets, served same-origin by the www-edge Worker (survives Kamal deploys)	webpack build / deploy
`heatwave-call-recordings-production`	—	Switchvox call recordings (`WarmlyYours/` prefix); Sidekiq imports from here	SFTPGo `pbx` user
`heatwave-pbx-backups-production`	ENAM	Switchvox PBX system backups (bucket root)	SFTPGo `pbx-backup` user

R2 bucket location is pinned on first creation of a name — delete+recreate reuses the original location (--location enam only applies to a fresh name). Tokens are minted via script/setup_r2_* and stored in 1Password.

Edge & network protection (what protects it)

Ingress: Cloudflare Load Balancing → two distinct outbound-only Dallas cloudflared tunnel UUIDs (two auto transport replicas each, normally QUIC) → kamal-proxy :80. No inbound web port is open on any host. A 30-second sibling-gated watchdog heals degraded connectors; checksum-verified updates roll through the secondary pair and then the primary pair. TLS terminates at Cloudflare (proxy.ssl: false); the www-edge Worker fronts www/apex for locale redirects, R2 webpack-asset serving, and cache rules.
Firewall — three layers:
1. Latitude edge firewall — allow SSH 22 from 100.64.0.0/10 (Tailscale) and SFTP 2222 from 144.202.57.170 (PBX); deny the rest.
2. Host UFW — default deny incoming; allow lo, tailscale0, 22/tcp, and 2222 from the PBX IP.
3. DOCKER-USER iptables (/usr/local/sbin/docker-user-fw.sh) — drops public 80/443, allows the tailnet, and handles the SFTP DNAT gotcha (match conntrack --ctorigdstport 2222, not --dport, because Docker DNATs 2222→2022 before the rule sees it).
Cloudflare Access: staging only — crm/www/api/mcp.warmlyyours.ws are gated to the wy-employees group (24 h session). Production has no CF Access by design (public site; Rails handles its own auth). The internal docs.warmlyyours.dev portal is gated to @warmlyyours.com.
Tailnet: all operator surfaces (SSH, Netdata, HAProxy stats, SFTPGo UI, Databasus, Mailpit, psql) are Tailscale-only. Stable VIPs heatwave-db (→primary) and heatwave-db-ro (→standby) survive CHI↔DAL flips. The Tailscale ACL is Terraform-managed (infra/terraform/tailscale/).

Infrastructure as code (Terraform Cloud)

All infra is OpenTofu/Terraform under infra/terraform/, applied by HCP Terraform (org warmlyyours), VCS-driven from this repo (auto-apply off — plans are reviewed, applies are manual).

Workspace	Module	Manages
`heatwave-latitude-production`	`latitude/`	The Latitude bare-metal server + cloud-init + per-host edge firewall
`heatwave-host-config`	`host-config/`	Re-runs the idempotent `provision-host.sh` over SSH (postfix relay, pg-maintenance/logwatch timers, ZED) — no reinstall
`heatwave-tailscale`	`tailscale/`	Tailnet ACL + the `heatwave-db` / `heatwave-db-ro` VIP services
`heatwave-cloudflare-zone-{production,staging}`	`cloudflare-zone-*/`	CF zone rulesets (WAF, cache, transforms) — the IaC source of truth; the dashboard is read-only

TFC agent: a hashicorp/tfc-agent container on Chicago (--network host, pool heatwave-tailnet) runs the agent-mode workspaces’ SSH provisioners — the one thing a hosted runner can’t do to a tailnet-only host. Dials out to app.terraform.io only. It is re-established by re-running the container (token op://IT/TFC-agent-token (heatwave-tailnet)).
⚠️ Latitude reinstall landmine (now guarded): a user_data change on latitudesh_server.host triggers a full server reinstall (all host data lost). On 2026-06-15 an approved plan whose only effective diff was a provision-host.sh edit reinstalled the live Chicago standby — wiping the PG standby + the Databasus controller and killing the in-flight TFC agent (it runs on Chicago). latitude/main.tf now carries lifecycle { ignore_changes = [user_data, billing] }, so editing cloud-init / provision-host.sh is inert w.r.t. a running box (day-2 host config flows through host-config over SSH). A genuine rebuild is now explicit: tofu apply -replace=latitudesh_server.host. Full record: doc/tasks/202606151240_CHICAGO_REINSTALL_DR_RECOVERY.md.

Application observability

Tool	Status	Notes
AppSignal	Live	Current APM / exception / host-metric sink (`Heatwave/production` + `/staging`)
Netdata	Live	Per-second infra metrics; per-host agents (not a parent/child stream)
PgHero	Live	Rails-mounted PG performance dashboard

ZFS pool DEGRADED/FAULTED alerting is handled by ZED (ZFS Event Daemon) over the postfix→SendGrid relay, not Netdata (the container→host firewall blocks zpool collection).

Staging (summary)

Staging co-locates on the Dallas box under the heatwave-staging- service prefix. It mirrors the full prod topology (primary + same-host standby for HAProxy-failover rehearsal, Valkey ×3, pgbouncer, HAProxy), but every port binds 127.0.0.1 (except Mailpit/Netdata on the tailnet, since prod owns the tailnet :19999). Staging hostnames crm/www/api/mcp.warmlyyours.ws are behind Cloudflare Access (wy-employees). Staging sends no real mail — it’s captured by Mailpit (see the admin-services table above).

Keeping this current

When config/deploy.yml / deploy.staging.yml, config/netdata/, config/haproxy/, or infra/terraform/ change, update this page. The facts here were extracted from those files on the date above; they are the source of truth, this is the index.