Menard inventory upload outage — 2026-07-13 → 2026-08-03

Three weeks of silent non-delivery of Menard inventory feeds, caused by an
incidental gem bump. Every layer that failed got a guard; this doc is the
map back if anything similar recurs.

Symptom

CRM: EDI Communication Log 3632647 (partner: menard,
category: inventory_advice) stuck in exception with
"Failed to download spreadsheet for upload" — a red herring (a transient
truncated Wasabi download on 08-03).
The real damage: 72 inventory snapshots piled up in ready, nothing
reached the Menard portal after 07-13 05:04, and no alert fired because
the job never errored — it hung until the next Sidekiq restart killed it.
The portal's "Last updated: 07/13/2026" confirmed it.

Root cause

Commit 8bcaa9bfef (07-13 10:54, "blog: apply consistent spacing…")
incidentally bumped playwright-ruby-client 1.60.0 → 1.61.0 in
Gemfile.lock and deployed between the 05:04 (success) and 13:00 (first
hang) runs. Gem 1.61.0's websocket reader thread dies on the PingOne
My Apps page:

Playwright::Connection#dispatch: Cannot find command to respond: 11 (RuntimeError)

Once that thread is dead nobody reads the socket, so every pending protocol
call — including its own server-enforced timeout replies — blocks forever.
No exception, no timeout, no ECL state change. Each scheduled run re-tried
the same oldest ready ECL, hung, and was eventually killed by a deploy
restart, wedging the queue indefinitely.

Playwright client/server pairs are version-sensitive: the server was
v1.60 until 07-18, then v1.61.1, while the gem sat on the buggy 1.61.0
throughout.

Fix (all shipped 2026-08-03)

Version lockstep restored — gem 1.62.0
(COMPATIBLE_PLAYWRIGHT_VERSION = 1.62.1) + accessory image/cmd
v1.62.1 in config/deploy.yml and config/deploy.staging.yml.
Accessory changes need kamal accessory reboot playwright — a normal
deploy does not touch accessories.
Gem pinned exactly in the Gemfile so bundle update cannot move it
without an intentional edit.
Lockstep enforced by CI —
test/services/transport/playwright_portal_connection_test.rb asserts
the gem's COMPATIBLE_PLAYWRIGHT_VERSION equals both the accessory
image tag and the npx playwright@… CLI version in both deploy files.
Bumping any one alone fails the build.
Sender watchdog — Edi::Menard::InventoryMessageSender wraps the
portal upload in Timeout.timeout(300); a hang now marks the ECL
exception (with an "upload hung" note + EDI admin email) instead of
wedging the queue for weeks.
Snapshot supersede — each Menard ECL is a full inventory snapshot,
so queue-mode sends only the newest ready ECL and archives older ones
as Superseded by ECL <id> (ready → archived transition added).
Restock-date clamp — the Menard portal rejects the entire upload
if any "Date when back in stock" is before the current date, and an
overdue purchase order put its past expected date straight into fresh
files. InventoryMessageProcessor#restock_info now clamps to tomorrow.
Poll-based URL waits — the transport polls page.url (client-side,
cannot hang on a dead reader thread) instead of protocol-level
wait_for_url, raising into the HTML-dump rescue on deadline.

Generalised (2026-08-04)

The fixes above are Menard-shaped; the failure is not. Every guard was raised
to the layer that all partners share:

Session cap on every browser flow —
PlaywrightRuntime.with_browser now wraps each session in
Timeout.timeout(SESSION_TIMEOUT_SECONDS), so the crawler, article→PDF,
and Redactor email render are covered by the same guard the Menard sender
got, instead of three more copies of it. Unwinding is safe:
WebSocketTransport#stop closes the socket with its own 2s bound. nil
and 0 are rejected — Timeout.timeout treats both as "no timeout".
lock_ttl on every EDI execute_flow worker — these walk all
orchestrators sequentially under a global
lock: :until_and_while_executing + on_conflict: :log + retry: 0, so
until now a run that HUNG suppressed every later scheduled run with nothing
but a log line. 2h is a ceiling (a real run takes seconds), not an SLA.
See "The overlap this buys" below — it is a deliberate trade, not an
oversight.
Diagnostics are bounded and log last — dump_page_html /
capture_screenshot run on an already-broken session, so each is capped at
DIAGNOSTIC_TIMEOUT, and logger.error now runs before the dump. A hung
diagnostic no longer eats the real error message.
EdiStalledFeedSweep — hourly, alerts per partner+category on ECLs
stuck in ready (6h from creation) or processing (26h from state entry,
past Walmart's own 24h abandonment). This is the detection layer whose
absence let both incidents below run for weeks; it does not care why a
sender stopped.
bin/deploy compares running accessory images to the config —
check_accessory_image_drift, warn-only. The lockstep unit test pins the
gem to config/deploy*.yml; only the deploy host knows the container was
still on v1.60.0 for 16 days after the config said v1.61.1.

Fan-out: inventory and price, 2026-08-04

EdiInventoryFlowWorker and EdiPriceFlowWorker are now dispatchers. Each
builds a Sidekiq Pro batch of one EdiOrchestratorFlowWorker per orchestrator
CLASS, with EdiFlowFinalizer on :complete — the wiki's serial → parallel →
back-to-serial shape, already proven by EdiProductDataFlowWorker.

Two things it buys:

Isolation. The unique lock is now per (flow, orchestrator), so a wedged
Menard run leaves Wayfair's next run free. lock_ttl stops being the only
thing between us and another multi-vendor outage.
A consumer for the results. execute_flow builds a per-partner result
array with :error entries and returned it to a Sidekiq worker, which
discarded it. EdiOrchestratorFlowWorker raises when any partner errored so
the batch counts it, and the finalizer reports it.

Per CLASS, not per partner — deliberately. Each vendor paces itself
internally (Amazon sleeps between marketplaces, and its twelve share two SP-API
accounts metered per ACCOUNT). Splitting partners into parallel jobs discards
that pacing and would make the EU starvation above worse, not better. One job
per vendor keeps every vendor's sequencing exactly as it was and still isolates
vendors from each other.

Still open, in order:

Spread Amazon's partners across the hour. This is what actually makes the
flow quota-compliant; FeedSubmissionBudget is only a burst ceiling. Needs
per-partner scheduling, not more parallelism.
Order flow. Not fanned out: it is a per-partner pipeline (retrieve →
process → acknowledge → confirm) whose sleep(1)s are anti-hammer pacing,
and it runs every 15 minutes. Check EdiCommunicationLog dedup on
order_batch / order_acknowledge before touching its concurrency.
Peak memory. The fanned-out jobs share the default queue, so N
orchestrators can now be in flight where one was before. If peak RSS becomes
the constraint the upgrade is a dedicated queue on the sidekiq_edi role —
a capacity change with its own DB_POOL/PgBouncer review.

The overlap `lock_ttl` buys, and why we took it

A TTL that expires while the previous run is still alive lets the next
scheduled run start alongside it. That is the point — "one overlapping run"
beats "no runs until someone notices" — but it is only reachable when a run
exceeds two hours, which now requires a hang outside the guarded browser
path.

An advisory lock spanning the flow (the EdiProductDataFlowWorker
DISPATCH_LOCK pattern) would remove the overlap, and was rejected: a hung job
holds its DB session, so the lock is never released and we are back to the
original bug — one wedged partner blocking the flow indefinitely, which is the
failure this whole document is about.

What overlap costs per flow, if it ever happens:

inventory / price / listing-feed — each message is a full snapshot, so a
duplicate send is wasteful, not wrong. Menard supersedes older ready ECLs;
Walmart's inventory_feed_in_flight? skips while a feed is ingesting.
discontinue — DELETE by SKU, idempotent.
order — the one to watch. Retrieval + acknowledge are not obviously safe
to run twice, which is why order flow is still not fanned out. Before raising
its concurrency, check EdiCommunicationLog dedup on order_batch /
order_acknowledge.

Amazon EU feed starvation — diagnosed 2026-08-04, fixed

amazon_seller_central_nl / _pl / _se had inventory_advice AND
price_advice ECLs piling up in ready since 2026-06-16 — 90+ each,
notes NULL, updated_at == created_at. BE/IT/FR/DE/ES/US/CA on the identical
flow processed normally.

It was never a hang. Every stuck row carried a transmit_after, and the
only writer of that column on this path is the 429 handler in
Edi::Amazon::FeedMessageSender. SP-API meters createFeed per seller
account, not per marketplace, and our twelve marketplaces share two:
:amazon_sc_seller_api (US/CA/MX) and :amazon_sc_seller_eu_api (the nine EU
ones). The partner loop spent the EU account's allowance on the first five
marketplaces and 429'd the rest — the same three, every run.

It was self-sustaining. Each hourly run rebuilt a fresh full snapshot and
replayed every superseded one still sitting in ready, so ~93 stale July
snapshots per partner consumed the account's budget before the current snapshot
was ever reached. A July 31 snapshot was still being retried on August 4.

Three changes:

FeedMessageSender#supersede_stale archives older ready messages for feeds
that carry the complete state, gated on a new full_snapshot_feed? that
defaults to false — inventory and price opt in; anything carrying less
than the whole picture is owed, not superseded. This is the dominant fix:
~93 submissions per partner per hour become 1.
Edi::Amazon::FeedSubmissionBudget paces submissions per account (fixed
window, Rails.cache) so a healthy run cannot burst nine marketplaces
through one account's allowance.
The 429 and budget-defer paths now WRITE why. They previously called
update_columns(transmit_after:) and nothing else, which skips updated_at
— so a rate-limited feed was byte-for-byte indistinguishable from a hung one.
That is the whole reason this hid for seven weeks.

The fingerprint, corrected

ready + NULL notes + updated_at == created_at is consistent with a
sender that stalled, never ran, or deferred — it proves only that nothing was
recorded. A hung sender, an unclaimed job, a scheduler gap, a worker outage and
(until this fix) an SP-API 429 all looked identical from the row. transmit_after
is the column that told them apart, and now notes does.

The one case it does rule out: a sender that raised would have written notes
and moved state.

Things learned the hard way

The Menard portal is IdP-initiated only: a direct goto to
partners.menard-inc.com/partners/greeting.html bounces back to PingOne
My Apps. The "Partners" tile click (page.expect_popup) is the required
flow — it was never the bug, gem 1.61.0 crashing on it was.
timeout: expected float, got undefined from the Playwright server =
client/server version mismatch (the accessory wasn't rebooted yet).
A Playwright hang leaves no trace: no exception, no AppSignal
incident. The tell in logs was "Login form rendered (PingOne SSO)"
followed by silence — no "Post-login URL" line.
Superseded snapshots go stale fast: a 3-week-old xlsx was rejected for a
past restock date even though it was valid the day it was generated.
Never re-send old snapshots; regenerate.