Kamal Stack — Troubleshooting Runbook

Symptom → cause → fix for the failure modes we’ve actually hit. Commands assume mise exec -- bundle exec kamal (abbreviated kamal), -d staging for staging. Box access is over Tailscale (ssh deploy@100.123.47.52).

---

Logs & troubleshooting — developer quick reference

Two prerequisites for every command below:

1. Be on Tailscale. Every box is reachable only over the tailnet — no public SSH or ports. tailscale status should list dal-latitude-heatwave-01 (Dallas — prod + staging, 100.123.47.52) and chi-latitude-heatwave-02 (Chicago — standby, 100.68.157.49).
2. Prefix kamal with the toolchain + pick the env: mise exec -- bundle exec kamal <cmd> targets production — it’s the base config/deploy.yml, so it takes no -d flag (-d production fails with Configuration file not found). Add -d staging for staging.

Looking at logs — there are four layers

Layer	Command	What you see
App (web + sidekiq)	kamal app logs -d staging -f	Rails/Puma + Sidekiq stdout (live). --roles=web / --roles=sidekiq to split; -n 200 for backlog; --grep ERROR to filter
Accessories (postgres / valkey ×3 / mailpit)	kamal accessory logs postgres -d staging -f (or valkey_cache / valkey_sessions / valkey_queue)	DB / cache / queue / mail-sink container logs
Edge connector (cloudflared, host systemd units)	Staging: ssh deploy@100.123.47.52 'journalctl -u cloudflared -n 50 -f'; production: ssh deploy@100.123.47.52 'journalctl -u cloudflared-prod -u cloudflared-prod-replica -u cloudflared-prod-secondary -u cloudflared-prod-secondary-replica -n 100 -f'	Tunnel up/down, QUIC/HTTP2 selection, ingress routing
Errors / APM	AppSignal apps Heatwave/staging & Heatwave/production	Exceptions, traces, slow requests — richer than tailing

Raw fallback on the box (when you need everything Docker sees): ssh deploy@100.123.47.52, then docker ps, docker logs <container> --tail 100 -f, docker stats. (Dallas/app 100.123.47.52, Chicago/standby 100.68.157.49.)

Troubleshooting flow

1. App or edge? Curl the proxy on the box, bypassing Cloudflare:

   ssh deploy@100.123.47.52 'curl -s -o /dev/null -w "%{http_code}\n" \
     -H "Host: www.warmlyyours.ws" -H "X-Forwarded-Proto: https" http://localhost/up'

   200 → the app is fine; the problem is at the edge (cloudflared / Cloudflare Access / tunnel). Anything else → app / proxy (next steps).
2. What’s running? kamal app details -d staging (+ docker ps on the box) — healthy? crash-looping? which image version (kamal app versions -d staging)?
3. Get inside: kamal console -d staging (Rails console), kamal shell -d staging (bash), kamal dbc -d staging (psql), or one-offs: kamal app exec -d staging --reuse 'bin/rails runner "…"'.
4. Specific symptom? Jump to the matching section below (healthcheck timeouts, versions partition 500s, matviews empty after a restore, OAuth redirect, uid-1001 asset EACCES, …).

The rest of this doc is the symptom-by-symptom runbook; the everyday-command table and day-2 ops live in MANAGING.md, the deploy flow in DEPLOYING.md, the architecture in README.md.

---

First-look triage

flowchart TD
    s{"What's broken?"}
    s -->|"Site won't load at all"| edge{"curl the box's kamal-proxy<br/>directly — does /up 200?"}
    s -->|"Deploy fails"| dep{"Where did it stop?"}
    s -->|"App up but errors / 500s"| app{"What kind of error?"}
    s -->|"Can't reach over Tailscale"| net["Check tailnet: tailscale status,<br/>use the 100.x IP, edge firewall :22"]

    edge -->|"yes (app fine)"| cf["Edge problem → check cloudflared<br/>+ Cloudflare Access / tunnel"]
    edge -->|"no"| ct["App/proxy down → kamal app details,<br/>docker ps, accessory boot order"]

    dep -->|"before build"| sec["1Password / clean-tree gate"]
    dep -->|"build"| bld["build secrets / .dockerignore / deps"]
    dep -->|"boot / healthcheck"| up["/up + force_ssl + config.hosts"]

    app -->|"DB error"| db["partitions / matviews / restore gaps"]
    app -->|"login blocked"| oauth["OAuth redirect URI"]
    app -->|"EACCES on assets"| uid["uid 1001 mismatch"]

A fast way to split “app vs edge”: curl the proxy on the box, bypassing Cloudflare:

ssh deploy@100.123.47.52 \
  'curl -sS -o /dev/null -w "%{http_code}\n" -H "Host: www.warmlyyours.ws" \
     -H "X-Forwarded-Proto: https" http://localhost:80/up'

200 = app healthy, problem is at the edge. Anything else = app/proxy problem.

---

Edge / ingress

“Site can’t be reached” / connection refused

• :3000 (or any non-standard port) in the URL. :3000 is Puma’s internal container port; Cloudflare only proxies standard web ports. The public URL has no port — https://www.warmlyyours.ws/en-US. The app never emits :3000 links itself. Fix: drop the port / clear the cached bookmark.

• Tunnel down. cloudflared runs as host systemd services, not containers. Staging still uses the distribution-created cloudflared unit; production uses two distinct Dallas tunnel UUIDs with two replicas each:

  # staging (Dallas)
  ssh deploy@100.123.47.52 \
    'systemctl status cloudflared; journalctl -u cloudflared -n 50'
  
  # production (Dallas)
  ssh deploy@100.123.47.52 '
    systemctl status cloudflared-prod cloudflared-prod-replica \
      cloudflared-prod-secondary cloudflared-prod-secondary-replica
    for port in 20242 20243 20244 20245; do
      curl -fsS "http://127.0.0.1:${port}/metrics" \
        | grep "^cloudflared_tunnel_ha_connections "
    done
    journalctl -u cloudflared-prod -u cloudflared-prod-replica \
      -u cloudflared-prod-secondary -u cloudflared-prod-secondary-replica -n 100
  '

  Healthy production = all four units active with 4 HA connections each and QUIC registered. cloudflared-watchdog.timer normally recovers an inactive or persistently degraded replica. For a manual restart, verify the sibling has four connections, restart one unit, wait for it to return to four, then touch the other; never restart both together.

  Automatic package updates use the site-specific timer, not Cloudflare’s vendor timer (which is masked):

  ssh deploy@100.123.47.52 '
    systemctl list-timers "cloudflared-update*"
    journalctl -u heatwave-cloudflared-update.service --since "30 days ago"
    sudo /usr/local/sbin/cloudflared-update --check-only
  '

  Dallas checks Tuesday after a seven-day release soak. The updater verifies APT/GitHub digests and rolls the secondary pair before the primary pair, always retaining a healthy same-tunnel sibling.

• Do not count Worker Cache API misses as customer 504s. The production www-edge Worker calls caches.default.match() before reading a webpack asset from R2. Cloudflare Analytics represents an internal Cache API miss as edge status 504, origin status 0, and requestSource=edgeWorkerCacheAPI; the outer browser request can still be a successful 200 with X-Served-By: r2-frontend-assets. Customer-impacting edge-error queries must filter to requestSource=eyeball (or at least group by request source) before alerting on 5xx totals.

• Edge is up but app is down. If the direct-proxy curl above is non-200, it’s not the edge — go to Deploy / boot.

Stuck at the Cloudflare Access login

A 302 to warmlyyours.cloudflareaccess.com is expected — staging is gated by the wy-employees Access group. Log in with a @warmlyyours.com identity. If a legitimate user is denied, check the Access policy/group in the Cloudflare Zero Trust dashboard (or infra/terraform/cloudflare/).

Google login blocked (redirect_uri_mismatch)

New staging hostnames aren’t in the Google OAuth client’s authorized redirect URIs. The CRM login uses Devise’s google_oauth2 provider; the callback is /accounts/auth/google_oauth2/callback. Staging has its own OAuth client (114261933316-b7694…, project “WY API Project”) — not prod’s. Add:

https://crm.warmlyyours.ws/accounts/auth/google_oauth2/callback

to that client’s Authorized redirect URIs in Google Cloud Console. (No JS origins needed — it’s a server-side code flow.) If the consent screen itself blocks, the registrable domain warmlyyours.ws may need adding to the consent screen’s Authorized domains (External apps) — Internal/Workspace apps are exempt.

---

Deploy / boot

1Password: “couldn’t connect to the 1Password desktop app”

The desktop-app CLI integration is flaky. In order of reliability:

1. ⌘Q the 1Password app fully (not just close the window), reopen, unlock. This clears a stuck CLI-integration helper and is the usual fix. Verify: op vault list. Confirm Settings → Developer → “Integrate with 1Password CLI” is ON.
2. Manual session in the deploy terminal: eval "$(op signin --account warmlyyours.1password.com)", then re-run.
3. Service-account token (most reliable — skips the desktop app entirely): save it to .kamal/.op-service-account-token (gitignored). See MANAGING.md → Secrets.

bin/deploy hard-gates on a real op read before the build, so a secret failure surfaces early with the exact op error.

“Working tree is not clean / out of sync”

The clean-tree gate refuses to ship code that matches no pushed commit (Kamal builds the working tree). Commit + push, or use --allow-dirty (throwaway staging test) / --in-worktree (clean master). See DEPLOYING.md.

Build failures

Error	Cause → fix
sidekiq-pro 401 during bundle install	BUNDLE_GEMS__CONTRIBSYS__COM didn’t resolve. Validate kamal secrets print -d staging; check op://IT/Sidekiq-Pro/credential.
Registry push 401	GHCR auth is per-developer via the gh CLI — there is no shared PAT (.kamal/secrets-common uses $(gh auth token)). One-time grant: gh auth refresh -h github.com -s write:packages, then bin/setup --doctor to verify. With a REMOTE builder, KAMAL_REGISTRY_* only authenticates the server-side PULL — the push uses your own local docker login ghcr.io. heatwave-build-cache is a separate package with its own access list.
db/structure.sql doesn't exist	.dockerignore stripped a needed file — un-ignore it.
vips: not found / a CLI missing at runtime	Add the package to the final stage’s apt list in the Dockerfile (e.g. libvips-tools).
webpack rebuilds from scratch every deploy	Known cache-invalidation issue — see doc/tasks/202606051345_WEBPACK_ASSET_BUILD_SPEEDUP.md (cache mount + revision decouple; likely folded into the Vite migration).

New container never goes healthy (/up times out, proxy gives up)

• /up returns 403. config.hosts allow-listing or force_ssl is rejecting the internal health probe. /up must be exempt from both — confirm the exemption in config/environments/*.rb. (This bit us on the first staging deploy.)
• Boot is just slow. Puma preload is ~20s; deploy_timeout is 90s. If a heavy boot legitimately needs more, raise deploy_timeout in config/deploy.yml.
• DB unreachable at boot. The app can’t boot without its DB. In staging, check the Postgres accessory is up (docker ps). In prod the app connects through the HAProxy write-VIP (DATABASE_HOST=heatwave-haproxy:6433) → the live primary’s pgbouncer → Postgres, so check that chain: is HAProxy showing a healthy backend (http://100.123.47.52:8404/), is the primary’s pg-health returning 200, and is the Postgres accessory up (Dallas primary, or Chicago standby after a flip)? See INFRASTRUCTURE_INVENTORY.md for the routing layout.

App crash-loops after a host reboot

Accessories and the app start independently, so the app may crash-loop until Postgres is ready. restart: unless-stopped retries it, so it usually self-heals. If not, boot the accessories first:

ssh deploy@100.123.47.52 docker ps                  # are postgres + the valkey trio up?
kamal accessory boot postgres        -d staging
kamal accessory boot valkey_cache    -d staging      # + valkey_sessions, valkey_queue
kamal accessory boot valkey_sessions -d staging
kamal accessory boot valkey_queue    -d staging
kamal app boot -d staging

Sidekiq stuck quieted after a failed deploy

pre-deploy quiets Sidekiq (TSTP); on a failed deploy bin/deploy un-quiets it automatically. If you ran bare kamal or the auto-resume failed:

kamal app boot --roles=sidekiq,sidekiq_heavy,sidekiq_edi -d staging

---

Database

PgBouncer was replaced and the site stayed down

Symptoms: PgBouncer is running again, but /up remains 502 or times out; Puma requests accumulate, AppSignal shows database/proxy errors, and restarting PgBouncer a second time does not recover the app.

Cause: PgBouncer uses session mode. A container replacement terminates the process behind every long-lived Active Record connection. Rails may reconnect after an ordinary connection error, but it is not safe to rely on that for a forced pooler replacement. On 2026-07-31 all 20 Puma request slots became blocked on unusable sessions; the site recovered only after web was recycled.

Immediate recovery: first confirm the new pooler and its Postgres backend are healthy. Then identify the application version that was live before the incident and explicitly pin the recycle to it—do not accidentally deploy the checkout’s current SHA:

mise exec -- bundle exec kamal accessory details pgbouncer
mise exec -- bundle exec kamal accessory logs pgbouncer -n 100
mise exec -- bundle exec kamal app versions

mise exec -- bundle exec kamal app boot --version <LIVE_VERSION> --roles=web
mise exec -- bundle exec kamal app boot --version <LIVE_VERSION> \
  --roles=sidekiq,sidekiq_heavy,sidekiq_edi

For staging, add -d staging to each Kamal command. Once web answers, verify CRM and WWW /up, all three Sidekiq processes and fetch state, queue latency, PgBouncer cl_waiting/maxwait, PostgreSQL active/idle-in-transaction sessions, and new AppSignal PG:: errors. Sidekiq Pro reliable fetch recovers work from a worker interrupted by this recycle.

Planned replacement: do not use the recovery sequence as the normal change procedure. Put the full application into the existing maintenance window so clients are stopped before the pooler disappears and boot with fresh connections afterward. Before stopping anything, confirm the target, verify the same current Git SHA in all four app roles, record it as <LIVE_VERSION>, check Kamal’s lock, explicitly coordinate that no wider bin/deploy pipeline is in progress, reserve the maintenance window, and obtain production approval:

mise exec -- bundle exec kamal app version
mise exec -- bundle exec kamal app details
mise exec -- bundle exec kamal lock status

bin/db maintenance up production
mise exec -- bundle exec kamal accessory reboot pgbouncer
bin/db maintenance down production <LIVE_VERSION>

Use staging, the confirmed staging revision, and -d staging for direct Kamal commands there. maintenance down pins the recorded image and returns nonzero without resuming the proxy when kamal app boot cannot health-gate web and all three worker roles. Do not overlap this with bin/deploy; see PgBouncer replacement safety and deployment concurrency.

Quote builder returns “No matching controls” (but prod is fine)

The view_quote_bom_items matview is empty. After a restore, matviews get refreshed during the schema-only phase (on empty base tables) and, if not re-refreshed against loaded data, stay empty. An empty BOM matview surfaces as “No matching controls” (the elements error is overwritten by the controls one in heating_system_items.rb). Fix:

kamal accessory exec postgres -d staging \
  "psql -U deploy -d heatwave -c 'REFRESH MATERIALIZED VIEW public.view_quote_bom_items;'"

The restore scripts now refresh this critical matview eagerly + verified pre-swap, so a fresh restore can’t reproduce it (see MANAGING.md → Database restore).

versions write 500s: “no partition of relation versions found”

PaperTrail’s versions table (separate heatwave_versions DB, FDW-backed) is range-partitioned by year. db/versions_structure.sql historically shipped with no child partitions, so the first write after a fresh load 500s. Fix: create the annual partitions + versions_default, then the structure dump carries them (the pg_party schema_exclude_partitions = false fix, config/initializers/355_pg_party.rb, PR #1031). On a box that’s already broken:

-- one-off, per missing year, in heatwave_versions:
CREATE TABLE versions_2026 PARTITION OF versions
  FOR VALUES FROM ('2026-01-01') TO ('2027-01-01');
CREATE TABLE versions_default PARTITION OF versions DEFAULT;

Analytics dashboards empty after a restore

The ~22 analytics matviews (view_sales_facts, view_opportunities_facts, view_visits_*, …) are refreshed after the swap (deferred, non-blocking) and self-heal via the hourly MatviewRefreshWorker cron. If staging’s scheduler is running they fill in on their own; to force it:

kamal app exec -d staging --reuse \
  'bin/rails runner "MatviewRefreshWorker.new.perform"'

Postgres collation mismatch on accessory boot

initdb collation (e.g. 2.41 vs 2.36) differs between the stock and custom images. The pgdata volume must be initialized under the custom image (ghcr.io/warmlyyours/heatwave-postgres:18, built from docker/postgresql.Dockerfile). If you see a collation-version warning, re-initdb the volume under that image — or, if the data is fine, ALTER DATABASE … REFRESH COLLATION VERSION. Relocating the image between registries does not change collation (the bytes are identical); a mismatch only appears if the pgvector/pgvector:pg18 base was rebuilt with a newer glibc, so prefer re-tagging an existing image over rebuilding when you just need to move registries.

---

App runtime

Sourcemap upload fails with EACCES (DELETE_MAPS)

The asset_path host volume is owned by the host deploy user (uid 1001); a container running as a different uid can’t rewrite/clean the bridged assets. The container app user must be uid 1001 (Dockerfile USER 1001, and cloud-init pins the deploy user to uid 1001). If you see EACCES on the post-deploy sourcemap cleanup, the uids have drifted — check both.

MissingKeyError: ENV['staging'] (or ['production'])

Heatwave’s Heatwave::Configuration is keyed on ENV['<environment>'], so each destination needs a staging / production secret = cat config/master.key (present in .kamal/secrets.staging / .kamal/secrets). Validate with kamal secrets print -d staging.

“Sunny is broken” / AI 400s

Unrelated to Kamal — that’s the AppSignal #3808 body-less Gemini 400 class. See the Sunny memory/notes, not this runbook.

---

Network / access

• Can’t SSH to the box. SSH is Tailscale-only (Latitude edge firewall allows :22 from 100.64.0.0/10 only). Confirm you’re on the tailnet (tailscale status) and using the Tailscale IP 100.123.47.52.
• psql / mailpit UI unreachable. Same — those bind to 127.0.0.1 (psql) or the Tailscale IP (mailpit :8025). They are never publicly exposed.
• A published container port is unexpectedly world-reachable. Docker bypasses UFW for published ports; the DOCKER-USER chain (cloud-init) is what blocks public :80/:443. Verify it loaded: sudo iptables -L DOCKER-USER -n.

---

Escalation references

• Migration status + cutover gates — doc/tasks/202606022303_KAMAL_MIGRATION.md
• DB-tier HA topology (Dallas primary / Chicago standby + HAProxy/pgbouncer) — doc/tasks/202606112045_DB_TIER_HA_ARCHITECTURE.md; live host/port reference — doc/infrastructure/INFRASTRUCTURE_INVENTORY.md
• Webpack build speedup — doc/tasks/202606051345_WEBPACK_ASSET_BUILD_SPEEDUP.md
• Errors / APM — the appsignal skill (AppSignal incidents, traces, logs)
• Cloudflare zones/tokens/workers — doc/infrastructure/CLOUDFLARE.md