Sidekiq operations and memory hardening

This runbook describes Heatwave's three Sidekiq application roles, their queue
coverage and connection budgets, the memory-recycle path, and the production
rollout checks. The implementation ledger and incident evidence are in
202607311447_SIDEKIQ_MEMORY_AND_WORKLOAD_HARDENING.md.

Why there are three application roles

The former worker was one 49-thread failure domain. When its custom memory guard
quieted the process, every queue stopped fetching for the entire graceful-drain
window. On 2026-07-31, AppSignal showed the process repeatedly growing to about
5.2 GiB and restarting while default and high queue latency climbed past 15
minutes. Elodie's mass-activity job itself ran in about 13 seconds after waiting
roughly 14 minutes for that process to fetch it.

Sidekiq workers run the Rails application and therefore remain Kamal application
roles, not accessories. They deploy and roll back with the same image revision.
The third role turns the host's available capacity into EDI throughput without
putting the latency and general-heavy failure boundaries back together.

Role and queue inventory

Role Queue source Queues Concurrency
sidekiq (SIDEKIQ_ROLE=latency) config/sidekiq.yml default, mailers, budgets, shipment_tracking, kit_consolidation 15
sidekiq capsules high (8), pbx (1), pbx_bulk (1), invoicing (1), online_migrations (1), Action Mailbox (1), Active Storage (1) 14
sidekiq_heavy (SIDEKIQ_ROLE=heavy) config/sidekiq_heavy.yml low, tracker, specs, api_heavy, amazon_api, ai_embeddings, images, pdf, data_heavy, low_priority 8
sidekiq_heavy capsules campaign (10), seo_visits (1) 11
sidekiq_edi (SIDEKIQ_ROLE=edi) config/sidekiq_edi.yml edi_product_data 4

The latency role has 29 worker threads and is the only scheduler owner.
Single-thread pbx and pbx_bulk capsules limit background traffic to at most
two concurrent calls to the legacy Switchvox server. Realtime presence and
current-call work cannot wait behind the 16–60 second import/reconciliation
jobs. Their threads came from high and the base capsule, so the role and
database budgets did not grow. The
heavy role has 19 worker threads and never loads the schedule. Its shared base
queues all have weight 1, putting that capsule in random mode so a sustained
low backlog cannot starve Amazon report polls or data-heavy dispatchers.
The EDI role has four worker threads, no capsules, and no scheduler. A Google
feed can issue five bounded Merchant HTTP requests at once, so the role can
drive at most 20 product inserts concurrently; HTTP waits do not consume an
Active Record connection.
Queue assignments are disjoint and guarded by
test/config/sidekiq_role_config_test.rb.

Memory budgets and recycle behavior

Role Warn Graceful recycle Docker hard limit Drain grace
sidekiq 6 GiB RSS 8 GiB RSS after full GC 12 GiB 120 seconds
sidekiq_heavy 6 GiB RSS 8 GiB RSS after full GC 12 GiB 900 seconds
sidekiq_edi 12 GiB RSS 18 GiB RSS after full GC 24 GiB 900 seconds

The guard runs after a job boundary. Only one capsule can claim a check or
restart for the process. Above the role's graceful threshold it runs one full
GC and recycles only if RSS remains above that threshold. It then quiets,
waits for active jobs, stops, and finally signals the process. Sidekiq Pro
super_fetch! recovers unfinished jobs after forced termination.

The latency/heavy roles have 4 GiB between graceful recycle and their cgroup
ceiling; EDI has 6 GiB because four feeds can be draining. The host's free
memory is useful only inside these isolated ceilings. A cgroup limit applies to
the whole container, while the guard measures the Ruby process's RSS, so the
two numbers intentionally differ.

Database connection budget

PgBouncer is in session mode, so configured pools represent real potential
server demand rather than multiplexed transactions.

Environment Web Latency worker Heavy worker EDI worker Steady total PgBouncer ceiling Unallocated
production 4 Puma processes × 7 = 28 38 20 5 91 92 per database 1
staging 4 Puma processes × 7 = 28 38 20 4 90 90 per database 0

The latency pool covers 29 worker threads plus up to eight Sunny tool permits
and one spare connection. The heavy pool covers 19 worker threads plus one
spare connection. The EDI pool covers four database-active jobs plus one spare
connection; its five-way outbound HTTP fan-out occurs after each bounded
record is loaded and does not multiply database demand. These are potential
client pools, not permanently open connections. A rolling deploy temporarily
has old and new clients; PgBouncer caps each database at the values above and
queues the overlap rather than allowing PostgreSQL to reject it. Watch
cl_waiting, maxwait, and ActiveRecord::ConnectionTimeoutError during the
rollout.

Capacity evidence and tuning rule

The production box measured 187 GiB RAM with about 105-106 GiB available, 32
logical CPUs, low load, and zero CPU, memory, or I/O pressure. PostgreSQL has
197 usable connections. Because both heatwave and heatwave_versions can
each reach PgBouncer's 92-connection production ceiling, their combined hard
limit is 184. This reserves 13 normal-user slots for pg-health, Netdata,
PgBouncer authentication, operators, and other direct clients.

Feed traversal is batched at 100 records. A production-shaped Google profile
for 50 products fell from 1,220 SQL queries to 19, and a 100-product comparison
produced equivalent payload structures. Google inserts use five bounded HTTP
requests per job. At measurement time, Merchant insert usage was 20,893/day
against quotas of 20,000,000/day and 60,000/minute.

Tune one dimension per full feed cycle. Raise EDI job concurrency by one only
when role RSS remains below 12 GiB, queue latency still warrants it, CPU
pressure and PgBouncer waiting remain zero, and partner quotas have headroom.
Any DB pool increase requires an explicit reallocation from another role,
intentional use of the one unallocated pooled slot, or a new
PostgreSQL/PgBouncer capacity plan. Optimize repeated queries and allocations
before adding threads to a feed that still scales poorly.

Production baseline after rollout

PR #1835 deployed on 2026-07-31; the following morning production revision
e34f830d48 was stable with all three worker roles present. This is a
point-in-time comparison baseline, not proof that a full daily workload cycle
has completed:

  • AppSignal reported exactly three Sidekiq processes for 110 consecutive
    minute samples. default, high, edi_product_data, and every other
    configured queue were empty; retries and overdue scheduled jobs were zero.
  • Container usage was about 2.0 GiB for latency under its 12 GiB cap, 1.45 GiB
    for heavy under 12 GiB, and 0.96 GiB for EDI under 24 GiB. No container had
    restarted or been OOM-killed after recovery.
  • Dallas still had roughly 97-99 GiB available, with zero CPU, memory, and I/O
    pressure and no swap activity. PostgreSQL showed 80 of 200 connections, three
    active sessions, no idle-in-transaction sessions, and no lock waits.
  • One amazon_api latency sample reached 35.9 seconds, below the 60-second
    warning. Default, high, and EDI latency stayed at zero.
  • RSS increased during the first several hours instead of remaining flat. The
    values were well below the new budgets, but the rollout stays open until a
    complete daily cycle distinguishes ordinary Ruby high-water growth from a
    repeatable workload leak.

AppSignal's built-in RSS is per process and can include shared pages. Do not
sum Puma or worker PID RSS and treat the result as container or host usage;
compare role-tagged trends with Docker cgroup usage and host pressure.

An empty live queue does not make dead jobs healthy. At this snapshot the dead
set was at its 9,999-entry cap. A read-only census found 9,951 entries with no
exception: they were duplicates sent there by sidekiq-unique-jobs' :reject
conflict strategy, not failed work. Thirty-eight exception-bearing embedding
entries carried the private-method defect addressed by PR #1837; a handful of
unrelated true failures remain separately triageable. Recent EDI nil logs were
also valid absence for schema-optional attributes rather than failed feed work.
These are correctness and retry-hygiene signals, not queue starvation.

Those counts are a 2026-08-01 point-in-time reading and had already drifted by
the following day. The census that actually governed the prune is recorded under
"Dead-set hygiene" below.

Dead-set hygiene

The global uniqueness lock remains :until_and_while_executing. Both sides use
on_conflict: :log: the accepted original remains queued or running, while the
duplicate is coalesced and logged without manufacturing a dead job. A conflict
means the lock did its job, so neither side raises.

Server conflicts previously used :raise, on the rationale that an unexpected
execution overlap should ride normal Sidekiq retry/exhaustion handling rather
than be dropped silently. That rationale did not survive contact with
retry: 0 in the same Sidekiq.default_job_options hash: with no retries the
raise went straight to the dead set — the outcome :reject was abandoned for —
and reported an AppSignal exception incident per worker class (six on 2026-08-01
alone: #6537, #6536, #6534, #6529, #6528, #6527). A worker that genuinely needs
an execution overlap surfaced can still override both retry: and its conflict
strategy.

A worker may override the lock or conflict strategy when its idempotency or
coalescing boundary differs, but :reject must not be used for ordinary
deduplication because it calls Sidekiq::DeadSet#kill without an exception.

Treat any census recorded here as evidence, not as a future deletion scope — the
composition churns against the 10,000-entry cap and drifts within hours. A blank
error_class alone is insufficient to prove a uniqueness conflict. Immediately
before any production pruning:

  1. Run and display a fresh read-only census with the exact Ruby/Redis filter.
    Constrain it to an explicit worker allowlist and a creation/failure window
    bounded to revisions before this change; require blank error_class and
    error_message plus the sidekiq-unique-jobs digest/conflict metadata. Exclude
    every payload whose provenance is ambiguous, and group the count by worker
    and revision window for review.
  2. Record the proposed deletion code, a restricted export/checksum of the exact
    selected payloads as the recovery artifact, retention for that artifact, and
    the blast radius. This is delete-only: external API cost and quota impact are
    zero. Retrying any cohort is a different bulk operation whose per-provider
    calls and quota cost must be counted first; never retry these stale
    duplicates merely to empty the dead set.
  3. Obtain the first explicit confirmation for that fresh count and plan. Then
    ask a separate scope/window question (worker allowlist, revision cutoff, and
    maintenance window), re-run the count with the answer, and obtain a second
    explicit confirmation before executing.

Exception-bearing jobs always stay outside this cohort for root-cause review
and deliberate, separately approved retry or deletion.

That protocol was executed on 2026-08-02. The prune is done — do not re-run
it. The census taken immediately before deletion found 5,829 conflict
duplicates and 4,170 exception-bearing entries, not the 9,951 and 38 recorded a
day earlier. Nothing was pruned in between; the set simply kept churning against
its cap while the EmbeddingWorker defect was still firing. This is why a fresh
census is mandatory rather than advisory.

Selection required all eight of: blank error_class, blank error_message, no
retry_count, no failed_at, on_conflict == "reject", a present
lock_digest, a present lock, and a dead-set score before
2026-08-01 15:15:15 UTC — the 3bd41d3e91 commit that stopped the server side
raising. on_conflict is stamped into the payload at enqueue and the server
middleware reads the strategy from the payload rather than from current
configuration, so "reject" in a payload is self-proving provenance. No entry
was ambiguous.

The result was 9,999 to 4,170: 5,829 removed across 41 workers, with no leftover
:reject duplicate. Every survivor carries an exception, 4,142 of them the
EmbeddingWorker NoMethodError cohort that PR #1837 fixed and the nightly
TextEmbeddingPopulationWorker backfills then re-embedded unaided. The recovery
artifact is a gzipped JSONL of the exact payloads, checksummed over the sorted
selection and retained until 2026-10-31 on the operator's machine — not in the
repository and not in the container.

That cohort cannot regrow: the default has been {client: :log, server: :log}
since 3bd41d3e91. A large new dead-set population is a genuinely new cause and
must be diagnosed as one rather than pruned on this precedent.

AppSignal dashboard and alerts

The production dashboard is Sidekiq workload hardening
(6a6ca7a41bc97de4f231ff7b). It includes:

  • latency for default and high;
  • built-in Sidekiq process RSS plus role-tagged memory-guard RSS;
  • restart requests, rolling restart count, and confirmed process starts;
  • P95 job duration and process-RSS delta by role, queue, worker, and outcome;
  • EDI partner duration/RSS delta, SEO aggregate progress, and Amazon Ads poll
    progress.

Active production triggers:

Trigger Condition ID
Default latency warning default > 60 seconds for 5 minutes 6a6ca7dfea1502a0c09c122e
Default latency critical default > 5 minutes for 2 minutes 6a6ca7df43985bb2b231ff7c
High latency warning high > 60 seconds for 5 minutes 6a6ca7dfa6a1490ce09c122e
High latency critical high > 5 minutes for 2 minutes 6a6ca7df2a2c52902a31ff7e
Latency RSS warning latency role > 6 GiB for 5 minutes 6a6d077501301f2d4131ff7a
Latency RSS critical latency role > 8 GiB for 1 minute 6a6d07849eae05d54e31ff7b
Heavy RSS warning heavy role > 6 GiB for 5 minutes 6a6d078946353bb6ae9c122c
Heavy RSS critical heavy role > 8 GiB for 1 minute 6a6d078fbf5cfd0b019c122c
EDI RSS warning EDI role > 12 GiB for 5 minutes 6a6d07946d1b34bf539c122f
EDI RSS critical EDI role > 18 GiB for 1 minute 6a6d07999eae05d54e31ff80
Per-process RSS backstop any Sidekiq PID > 12 GiB for 5 minutes 6a6d07ac20bd29cc9631ff7a
Per-process RSS critical backstop any Sidekiq PID > 18 GiB for 1 minute 6a6d07b17ec73466639c122f
Repeated memory restart one role requests more than one restart in an hour 6a6ca8683745a4d9d331ff7b

The built-in process_rss metric is reported in KiB. The alert thresholds are
therefore 12,582,912 and 18,874,368 for the continuous PID backstops. The
custom sidekiq.memory_guard.rss_mb metric is reported in MiB and is grouped
by role, so its thresholds are 6,144/8,192 or 12,288/18,432. The former
process-name-wide 6/8 GiB triggers are retired after the three-role rollout;
they would incorrectly page on healthy EDI headroom.

Job RSS delta is candidate attribution, not proof: other threads can allocate
while a job runs. Prefer repeatable role and stage-correlated metrics plus
profiling before changing a worker.

Triage and restart runbook

  1. Confirm which role is affected and whether it is still fetching:

    mise exec -- bundle exec kamal app details
mise exec -- bundle exec kamal app logs --roles=sidekiq -n 300
mise exec -- bundle exec kamal app logs --roles=sidekiq_heavy -n 300
mise exec -- bundle exec kamal app logs --roles=sidekiq_edi -n 300

  2. Check the dashboard for queue latency, role RSS, restart_requested, drain
    duration, and the worker/stage immediately preceding the rise. Check the
    Sidekiq Web UI for busy jobs, retries, and scheduled jobs.

  3. If a failed deploy left worker roles quiet, boot all current-version roles:

    mise exec -- bundle exec kamal app boot --roles=sidekiq,sidekiq_heavy,sidekiq_edi

  4. If one role is unhealthy, restart only that failure domain. Do not restart
    the latency role merely because the heavy role is large:

    mise exec -- bundle exec kamal app boot --roles=sidekiq_heavy
# or, for EDI only:
mise exec -- bundle exec kamal app boot --roles=sidekiq_edi
# or, for the latency role only:
mise exec -- bundle exec kamal app boot --roles=sidekiq

  5. Verify the restarted role is fetching, its RSS reset, the other role stayed
    available, and PgBouncer has no sustained waiting clients.

If a release caused the regression, list versions and roll the whole application
revision back so web and all three workers remain on matching code:

mise exec -- bundle exec kamal app versions
mise exec -- bundle exec kamal rollback <VERSION>

A rollback also runs the pre/post-deploy hooks. Never use a schema rollback as
the first response; database rollback is separately gated because it can revert
data.

First SEO aggregate rollout

The initial 90-day aggregate writes more than 1,000 rows on production-scale
data. The scheduled worker therefore returns initial_backfill_gated until an
operator passes allow_backfill: true. Normal runs automatically refresh only
the latest two days after coverage is complete.

Before the one-time enqueue:

  1. Run an exact count against production for the 90-day window: matching source
    visits, distinct normalized date/path aggregate rows, and current SiteMap rows
    that the apply phase will update. Use the same Chicago calendar bounds and
    path normalization as Seo::VisitPageDailyCountsQuery.

  2. Surface the SQL/Ruby scope and actual counts for confirmation one.

  3. Ask a separate scope/blast-radius question. This operation uses only the
    database—there is no billed or rate-limited external API cost.

  4. Obtain confirmation two, then enqueue exactly one approved run:

    mise exec -- bundle exec kamal app exec --primary --roles=web --reuse \
  'bin/rails runner "SeoVisitsSyncWorker.perform_async({ allow_backfill: true })"'

Each date is transactional, protected by an advisory lock, and marked complete
even when it has zero matches. An interruption resumes only missing dates. The
apply phase starts only when every daily job succeeds and updates SiteMaps in
500-row batches. The aggregate tables are derived data and can be rebuilt; do
not delete their rows or prune queued jobs without repeating the bulk-operation
safety check for the exact cleanup scope.

BuildKit memory ceiling

The remote BuildKit container shares the application host. .kamal/hooks/pre-build
creates a missing docker-container builder with a 16 GiB limit or applies that
limit to the existing builder before every build. memory-swap=16g means no
additional swap allocation. The hook aborts if it cannot verify both limits.

Read-only verification:

ssh deploy@100.123.47.52 \
  "docker inspect buildx_buildkit_kamal-remote-ssh---deploy-100-123-47-520 \
   --format '{{.HostConfig.Memory}} {{.HostConfig.MemorySwap}}'"
# expected after the first build with the hook: 17179869184 17179869184

Rollout checklist

  • Validate production and staging Kamal config and all focused tests.
  • Deploy during an observed window without separately booting a datastore
    accessory.
  • Confirm sidekiq, sidekiq_heavy, and sidekiq_edi are all running the
    same revision.
  • Confirm one scheduler loaded, every queue has its intended consumer, and
    default/high remain below 60 seconds.
  • Observe at least one complete EDI product-data cycle and Amazon report
    transition.
  • Complete the separately approved initial SEO aggregate and verify it
    reaches 90 durable date markers before the apply phase.
  • Keep the rollout open until role RSS, restart frequency, DB headroom,
    retries, and completion rates remain healthy through a full daily cycle.

The first four gates were verified on 2026-08-01. The remaining gates are
deliberately open; a quiet early-morning snapshot is not a substitute for the
daily EDI, SEO, and advertising workload.