Class: Retailer::ProbeAutoSkipper

Inherits:
Object
  • Object
show all
Defined in:
app/services/retailer/probe_auto_skipper.rb

Overview

Detects catalog items whose retailer URL has been failing repeatedly and puts
them on a timed probe backoff, so the daily BatchPriceChecker stops
billing Oxylabs to re-probe a dead/wrong URL every day.

== Backoff, not skip, and never a state change

This used to write skip_url_checks, which is the HUMAN's "never probe this
listing" switch. Borrowing it had two costs. An active item ended up
claiming to be sellable while opting out of the check that proves it, and
because nothing ever cleared the flag, a retailer outage retired listings
permanently and silently — Ferguson Home lost 239 live listings, every URL and
price still correct. The backoff expires on its own, so an item that recovers
is picked up again with no human involved and active keeps meaning probeable.

The pause escalates 24h -> 48h -> 72h and then STOPS escalating. An active
catalog item is a standing claim that we are selling something, so that claim
has to stay falsifiable: the longest we ever go without asking is three days.
An item can be probed less often; it can never be switched off. Once the ladder
is exhausted the item still gets probed every 72h and additionally raises a
needs_manual_confirmation listing issue, because nothing the system can
measure separates "blocked" from "delisted" and only the second is a
merchandising decision.

It deliberately does NOT move the item's state either. needs_onboarding
reads like the right verb — "live but not listed at the retailer" — but
pending_onboarding is absent from CatalogItem::ORCHESTRATOR_STATES, and
omission from an orchestrator feed is a REMOVAL at the marketplace. Ten days
of a blocked scraper would delist us from Best Buy.ca, eBay, NewEgg, Sears and
Leroy Merlin (the ADEO feed outage, AppSignal #6521, was this exact mechanism).
A failing probe means "we could not verify", which is not the same claim as
"this listing is gone", and only the second one justifies touching state. The
not_confirmable listing issue is where a human makes that call.

Why this exists: when a probe finishes with status failed, not_found, or
product_mismatch, no retail_price is captured. SiblingPriceRefresher
treats such items as perpetually stale and re-probes them on every Amazon
pricing run, and the next nightly batch probes them again. Audit (May 2026)
showed ~9% of monthly Oxylabs volume was wasted on items that had never
returned a successful price in the last 30 days.

Call this once after saving a probe. Idempotent and cheap (one indexed
query against (catalog_item_id, created_at)).

Examples:

probe.save!
Retailer::ProbeAutoSkipper.maybe_skip!(catalog_item)

Constant Summary collapse

PROBE_WINDOW =

How many of the most-recent probe DAYS to evaluate.

10
BACKOFF_STEPS =

Escalating pause, and then a ceiling. Each consecutive trip moves one rung
down the ladder; past the last rung the pause STAYS at 72h forever.

The ceiling is the whole design. An active catalog item is a standing claim
that we are selling something, and that claim has to remain falsifiable, so
the longest we will ever go without asking is three days. An item can be
probed less often. It can never be switched off.

[24.hours, 48.hours, 72.hours].freeze
FAILURE_THRESHOLD =

Auto-skip once at least this many of the last PROBE_WINDOW days failed to
yield a price. This is a failure rate, not a strict consecutive streak: a
retailer that blocks our scraper but still succeeds intermittently would
keep resetting a consecutive streak and never trip. rona.ca (May 2026)
began timing out / faulting ~85% of Oxylabs probes while a handful still
succeeded — the old all-consecutive rule never fired. See the
"Heatwave retailer-probe budget" section of .agents/skills/oxylabs/SKILL.md.

8
NON_SUCCESS_STATUSES =

Statuses that count as "non-success".

%w[failed not_found product_mismatch].freeze

Class Method Summary collapse

Class Method Details

.exhausted?(catalog_item) ⇒ Boolean

The item has spent the whole ladder failing, so the system has nothing left
to try: it cannot tell a blocked scraper from a delisted product, and only
the second is a merchandising decision. Surfaces as a
needs_manual_confirmation listing issue. The item KEEPS being probed every
72h — this is an escalation, not a stop.

Parameters:

Returns:

  • (Boolean) 


123
124
125
 
# File 'app/services/retailer/probe_auto_skipper.rb', line 123

def self.exhausted?(catalog_item)
  catalog_item.probe_backoff_level.to_i >= BACKOFF_STEPS.size
end

.maybe_skip!(catalog_item, window: PROBE_WINDOW) ⇒ Boolean

Inspects the catalog item's most-recent probe days; if at least
FAILURE_THRESHOLD of the last PROBE_WINDOW produced no successful probe,
moves the item one rung down BACKOFF_STEPS so batch and sibling-refresh runs
leave it alone until the pause expires. Never writes skip_url_checks.

Parameters:

  • catalog_item (CatalogItem)
  • window (Integer) (defaults to: PROBE_WINDOW) —

    lookback size in days (override for tests)

Returns:

  • (Boolean) —

    true if we just backed the item off; false otherwise



83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
 
# File 'app/services/retailer/probe_auto_skipper.rb', line 83

def self.maybe_skip!(catalog_item, window: PROBE_WINDOW)
  return false if catalog_item.skip_url_checks?
  return false if catalog_item.probe_backoff_until&.future?

  daily = daily_outcomes(catalog_item, window)
  return false if daily.size < window

  non_success = daily.count(false)
  return false if non_success < failures_for(window)

  level = [catalog_item.probe_backoff_level.to_i + 1, BACKOFF_STEPS.size].min
  step = BACKOFF_STEPS[level - 1]

  # Save WITHOUT running the model's full validations. A CatalogItem can carry
  # pre-existing invalid data unrelated to URL probing — e.g. a blank `amount`
  # (validated at catalog_item.rb:252) — and a plain `update!` would raise
  # RecordInvalid on it mid-webhook (AppSignal #6019). `save(validate: false)`
  # still fires callbacks, updated_at and PaperTrail versioning, which is the
  # audit trail of when the system backed off and how far down the ladder it
  # has got. `update_columns` would skip all three.
  catalog_item.assign_attributes(probe_backoff_level: level, probe_backoff_until: step.from_now)
  catalog_item.save(validate: false)

  ceiling = ' — AT CEILING, needs a human' if exhausted?(catalog_item)
  Rails.logger.warn(
    "[ProbeAutoSkipper] Backing off CatalogItem #{catalog_item.id} for #{step.inspect} " \
    "(rung #{level}/#{BACKOFF_STEPS.size}#{ceiling}) " \
    "after #{non_success}/#{window} days without a successful probe"
  )
  true
end

.record_success!(catalog_item) ⇒ void

This method returns an undefined value.

A probe came back with a price, so the item is healthy: drop it back to the
top of the ladder. Without this the rungs are one-way and a listing that
recovers still creeps toward the ceiling on its next bad week.

Parameters:



133
134
135
136
137
138
 
# File 'app/services/retailer/probe_auto_skipper.rb', line 133

def self.record_success!(catalog_item)
  return if catalog_item.probe_backoff_level.to_i.zero? && catalog_item.probe_backoff_until.nil?

  catalog_item.assign_attributes(probe_backoff_level: 0, probe_backoff_until: nil)
  catalog_item.save(validate: false)
end