cloudflared — Cloudflare Tunnel (post-Kamal)

How the Heatwave hosts receive inbound web traffic without opening a single inbound web port. Companion to INFRASTRUCTURE_INVENTORY.md — see its Edge & network protection and Fleet sections for the surrounding edge facts.

The pre-Kamal version of this file (hand-run cloudflared tunnel login / create / route dns against Vultr boxes, with a local config.yaml ingress and a Capistrano deploy model) was removed — it described an architecture that no longer exists.

What it is

cloudflared is the Cloudflare Tunnel connector. Production has two distinct tunnel UUIDs on the Dallas application host: a Load Balancer primary and secondary, each with two host systemd connector replicas. Each connector normally holds four outbound QUIC connections to Cloudflare. Transport mode is auto: QUIC is preferred, with HTTP/2 as the built-in fallback when UDP is unavailable. Cloudflare routes inbound HTTPS for the public hostnames back down the selected tunnel to kamal-proxy :80 on the same host.

Consequences of the outbound model:

No inbound web port is open on any box — DOCKER-USER iptables drops public 80/443. The tunnel is the only path in for web traffic.
TLS terminates at Cloudflare. Kamal runs proxy.ssl: false; the tunnel hands plain HTTP to kamal-proxy over the loopback.
Public hostnames crm/www/api/scan/mcp.warmlyyours.com (.warmlyyours.ws for staging) resolve to the tunnel via proxied *.cfargotunnel.com CNAMEs.

Installed by cloud-init, managed remotely (Terraform)

There is no tunnel config file on the host. The tunnel, its ingress rules, and the DNS records are declared in Cloudflare and managed by Terraform — prod in infra/terraform/cloudflare-production/, staging in infra/terraform/cloudflare/. Both create a cloudflare_zero_trust_tunnel_cloudflared with config_src = "cloudflare" (remotely-managed) plus a …_cloudflared_config ingress block (hostname → http://localhost:80, with the required http_status:404 catch-all) and the proxied DNS records.

Cloud-init installs the connector binary (infra/terraform/latitude/cloud-init.yaml.tftpl). Production then uses the reviewed host definitions under infra/terraform/files/:

cloudflared-prod.service — primary tunnel replica 1, metrics on 127.0.0.1:20242.
cloudflared-prod-replica.service — primary tunnel replica 2, metrics on 127.0.0.1:20243.
cloudflared-prod-secondary.service — secondary tunnel replica 1, metrics on 127.0.0.1:20244.
cloudflared-prod-secondary-replica.service — secondary tunnel replica 2, metrics on 127.0.0.1:20245.
cloudflared-watchdog.{service,timer} and cloudflared-watchdog — guarded liveness recovery every 30 seconds.
cloudflared-connectors.sh — the shared unit, metrics-port, sibling, and tunnel mapping consumed by the watchdog and updater.
heatwave-cloudflared-update.service, cloudflared-update, and the Dallas update timer — verified, staggered rolling upgrades without connector downtime.
cloudflared-production-install — the Dallas-only installer that writes both token files before enabling connectors, deploys the reviewed units/scripts, and verifies all four replicas.

The primary pair reads /etc/cloudflared/heatwave-prod.env; the secondary pair reads /etc/cloudflared/heatwave-prod-secondary.env. Both files are root-only (0600), and neither token appears in process arguments or unit files. A minimal first-boot install still looks like:

# cloud-init runcmd (only runs if cloudflared_token is non-empty):
dpkg -i cloudflared-linux-amd64.deb
cloudflared service install <connector-token>

The connector token is the tunnel_token output of the Cloudflare module, piped into the latitude module as var.cloudflared_token for a new host’s minimal first boot:

(
  set -euo pipefail
  cloudflared_token="$(
    tofu -chdir=infra/terraform/cloudflare output -raw tunnel_token
  )"
  test -n "$cloudflared_token"
  TF_VAR_cloudflared_token="$cloudflared_token" \
    tofu -chdir=infra/terraform/latitude apply
)

Production’s dual-tunnel units use the tracked installer instead. It consumes both sensitive HCP outputs on standard input, never command arguments, and creates the primary and secondary environment files as root:root 0600 before enabling either secondary unit:

# From the repository root. The staging directory contains no secrets.
ssh deploy@100.123.47.52 'install -d -m 0700 /tmp/heatwave-cloudflared'
scp infra/terraform/files/cloudflared-* \
  infra/terraform/files/heatwave-cloudflared-update.service \
  deploy@100.123.47.52:/tmp/heatwave-cloudflared/

export TF_CLOUD_HOSTNAME=app.terraform.io
{
  tofu -chdir=infra/terraform/cloudflare-production output -raw tunnel_token
  printf '\n'
  tofu -chdir=infra/terraform/cloudflare-production \
    output -raw dallas_secondary_tunnel_token
  printf '\n'
} | ssh deploy@100.123.47.52 \
  'sudo /tmp/heatwave-cloudflared/cloudflared-production-install \
    --source-directory /tmp/heatwave-cloudflared --tokens-stdin'

ssh deploy@100.123.47.52 'rm -rf -- /tmp/heatwave-cloudflared'

For later unit or script updates, stage the same reviewed files and use --preserve-tokens; the installer verifies and retains the existing root-only files. It refuses to run anywhere except dal-latitude-heatwave-01, holds the shared connector-maintenance lock, keeps Cloudflare’s vendor update timer masked, and waits for all four replicas to recover to four HA connections.

No cloudflared tunnel login / create / route dns is ever run by hand — those steps are Terraform resources. Production state is remote and locked in HCP Terraform workspace heatwave-cloudflare-production.

Production topology and failover

Both production tunnels and all four connector processes run on Dallas (dal-latitude-heatwave-01) and route to the same localhost:80 application origin. Chicago is the PostgreSQL standby and is not an HTTP Load Balancer origin.

The Cloudflare Load Balancing model is one Dallas-primary pool and one Dallas-secondary pool per public hostname. Tunnel endpoints require a hostname-specific endpoint Host override, so the five hostnames consume ten billed origin entries even though there are only two tunnel UUIDs.

Cloudflare’s Load Balancing subscription exposes 10 origin entries (the base two plus eight additional entries). The base allowance was insufficient and Cloudflare accepted a subset of concurrent pool creates before rejecting the rest, so always re-plan against the locked remote state after changing the entitlement or after any failed apply.

The cloudflare-production module manages both tunnels, ingress, monitors, pools, and the five same-name load balancers. var.activate_load_balancing defaults to true, which is the live steady state. Set it to false only for an intentional rollback: applying removes the load balancer objects and reveals the existing Terraform-managed proxied CNAMEs underneath them.

The secondary UUID is independently health-checkable by Load Balancing; two replicas inside one UUID would appear as only one endpoint. This topology protects the Cloudflare tunnel/control path and connector updates. It does not protect against loss of the shared Dallas application host.

Self-healing behavior

systemd uses Restart=always with a five-second delay for process exits. The 30-second watchdog covers the harder failure mode where a process remains alive but loses tunnel connections:

Four HA connections is healthy; fewer than four is degraded.
A degraded process must fail three consecutive checks before recovery.
The same-tunnel sibling must normally have at least two connections before a live process can be restarted. If both same-tunnel replicas are degraded, the watchdog may recover one only when at least one connector in the other tunnel is fully healthy.
Only one connector is restarted per run, with a five-minute per-unit cooldown and a 30-second recovery check.
Inactive units honor the same five-minute cooldown. The watchdog continues evaluating the other replicas after an attempt, but never attempts a second restart during that run.

State is held under /run/cloudflared-watchdog; decisions and recovery results are recorded in the system journal.

Automatic updates

The connector units deliberately keep --no-autoupdate. All four processes share /usr/bin/cloudflared, so independent in-process updaters could race while replacing the same binary or restart replicas without a sibling-health gate. A single host-level updater owns package changes instead.

The updater checks weekly; it only changes the package when Cloudflare publishes a newer release:

Dallas checks Tuesday at 04:00 UTC (plus up to 30 minutes of jitter) after a seven-day release soak.
The updater bootstraps Cloudflare’s stable APT repository with the current signing key pinned to its primary fingerprint. The candidate must exist in that signed repository and match the latest official GitHub release and its SHA-256 digest. A future signing-key rotation fails closed until the new key is reviewed and its fingerprint is updated in the script.
A package is rejected if its maintainer scripts can start, stop, or restart services; connector restarts remain under our control.
All four replicas must be active on four HA connections before installation. The secondary tunnel pair rolls first, followed by the primary pair. One same-tunnel sibling remains connected during every restart.
Failure at either step restores the previous package (or the preserved binary if the old package has left APT) and recovers one replica at a time. The watchdog and updater share a lock, so they cannot restart connectors concurrently. The updater waits up to two minutes for the watchdog; if that bounded wait expires it leaves the package unchanged and the next scheduled attempt is the following Tuesday.
Only the rollback binary and package for the version currently being replaced are retained; older artifacts are pruned after the new rollback pair is staged.
The site timer is persistent. A missed Tuesday run starts on the next boot via the service’s --timer mode, which bypasses only the weekday gate and still enforces the seven-day release soak.
Cloudflare’s vendor cloudflared-update.timer is masked. Only the site-specific timer may invoke the Heatwave updater.

Manual checks do not install anything:

sudo /usr/local/sbin/cloudflared-update --check-only
systemctl list-timers 'cloudflared-update*'
journalctl -u heatwave-cloudflared-update --since '30 days ago'

--run-now bypasses the weekday but retains the site’s release soak period. Use it only when an out-of-band release has already been reviewed; it still keeps every checksum, package, sibling-health, rolling-restart, and rollback gate.

Verify

On Dallas (over the tailnet):

systemctl status cloudflared-prod cloudflared-prod-replica \
  cloudflared-prod-secondary cloudflared-prod-secondary-replica
systemctl status cloudflared-watchdog.timer
systemctl status cloudflared-update-dallas.timer

curl -fsS http://127.0.0.1:20242/metrics \
  | grep '^cloudflared_tunnel_ha_connections '
curl -fsS http://127.0.0.1:20243/metrics \
  | grep '^cloudflared_tunnel_ha_connections '
curl -fsS http://127.0.0.1:20244/metrics \
  | grep '^cloudflared_tunnel_ha_connections '
curl -fsS http://127.0.0.1:20245/metrics \
  | grep '^cloudflared_tunnel_ha_connections '

journalctl -u cloudflared-prod -u cloudflared-prod-replica \
  -u cloudflared-prod-secondary -u cloudflared-prod-secondary-replica \
  --since '15 minutes ago' | grep -E 'Initial protocol|protocol='
journalctl -u cloudflared-watchdog --since '15 minutes ago'
journalctl -u heatwave-cloudflared-update --since '30 days ago'

All four metrics endpoints should report 4, all connector journals should show Initial protocol quic, and both tunnels should show two current connector instances in Cloudflare Zero Trust (Networks → Tunnels).