Class: ProblematicDeliverySweep

Inherits:
Object
  • Object
show all
Includes:
Sidekiq::Job
Defined in:
app/workers/problematic_delivery_sweep.rb

Overview

Hourly sweep that escalates problematic parcel deliveries to the assigned
sales rep (direct customers) / orders@warmlyyours.com (marketplace orders,
and as a catch-all cc) — see
doc/tasks/202607211802_PROBLEMATIC_DELIVERY_ESCALATION.md.

Evaluates recently shipped, not-yet-delivered parcel shipments (per
tracking number, NOT per order — the motivating Amazon order 112-5853649-
4121036 / DE793239 had one package delivered and one returned to sender)
against four criteria over the ShipmentEvent scan projection:

  • +exception+ — an EX/AT scan with no later delivered (DE) scan

  • +pickup_in_lieu+ — latest terminal scan is SP (delivered to a
    collection location) with no DE to the address
    A FedEx multi-piece shipment inherits its master tracking number's scans
    (see ProblematicDeliverySweep.consignment_tracking_number) — the pieces travel as one
    consignment, so a per-piece scan we never received is not a lost parcel.
    A Canpar non-lead box likewise inherits its lead box's scans — Canpar
    emits scans only under piece 001 of a consignment. UPS and everyone else
    are per-package.

  • +no_pickup_scan+ — we made the label and ShipEngine is tracking it,
    but NO_PICKUP_SCAN_GRACE later the carrier has
    never scanned it into custody (see
    ProblematicDeliverySweep.no_pickup_scan?)

  • +overdue_no_delivery+ — past estimated_delivery_date (+1 day grace) with
    no delivered scan at all, UNLESS the parcel is
    still scanning cleanly (see TRAIL_HEALTHY_WINDOW)
    — ShipEngine reports delivery late often enough
    that "no DE scan" alone is not evidence of trouble

This worker DETECTS; it no longer notifies. Each first sighting is recorded
as an Events::ProblematicDeliveryDetected, gated on a 7-day Rails.cache flag
per (shipment-or-delivery, reason) so the same problem isn't re-recorded
every hour while ops works it (the MissedFreightPickupSweep convention).
ProblematicDeliveryDigestWorker mails the roll-up once a day, and the
warehouse Problematic Deliveries tab reads ProblematicDeliverySweep.flagged live.

Scheduled hourly via sidekiq-scheduler — see
config/sidekiq_production_schedule.yml and
config/sidekiq_staging_schedule.yml.

Constant Summary collapse

LOOKBACK_WINDOW =

Don't scan the entire history — only recently shipped parcels. Matches
Shipment::TRACKING_LOOKBACK_DAYS (tracking polling stops after 30 days).

30.days
OVERDUE_GRACE_DAYS =

Days past estimated_delivery_date before an undelivered shipment counts
as overdue ("within the expected time frame +/- a day or so").

1
TRAIL_HEALTHY_WINDOW =

How long a clean, still-moving scan trail suppresses +overdue+.

ShipEngine pushes the delivered (DE) webhook long after the parcel
actually arrives: measured over 45 days of prod, 19.5% of DE webhooks
land more than 6h late with a p90 of 28h, while in-transit scans hold
a 1.2h p90. So "no delivered scan" frequently means "not reported
yet", not "not delivered" — and alerting on those trains ops to
ignore the tab. On 2026-07-30 that produced 17 overdue rows of which
one was worth chasing.

Sized on the gap from a parcel's last movement scan to the moment we
INGEST its DE scan, over 1,856 delivered parcels: 6.7h median, 11.5h
p90, 14.7h p95. Cumulative coverage is 96.8% at 24h, 98.0% at 36h,
98.3% at 48h — so 36h buys a real 1.2 points over 24h and 48h buys
almost nothing. Past 36h of silence, "not reported yet" stops being
the likely explanation and the parcel is worth a human look.

36.hours
FREIGHT_TRAIL_HEALTHY_WINDOW =

TRAIL_HEALTHY_WINDOW's freight counterpart. LTL scans are an order of
magnitude sparser than parcel scans, so 36h of silence is normal rather
than diagnostic: over 322 consecutive freight-event gaps the p50 is 0.4h
but the p90 is 23.6h, p95 57.3h and p99 153.6h. 96h covers 96.9% of
legitimate gaps (13 of 322 exceed 72h; 10 exceed 96h).

96.hours
CUSTODY_STATUS_CODES =

An LTL delivery has no estimated_delivery_date to be "past", and days
since pickup is a hopeless discriminator — measured pickup-to-delivered
over 21 completed loads is p50 3.9d, p90 11.0d, max 13.0d. Silence is the
sharp signal instead, so freight overdue is "in carrier custody, no
delivery event, and nothing scanned for FREIGHT_TRAIL_HEALTHY_WINDOW".

Requiring custody is what keeps this off MissedFreightPickupSweep's
territory: a load that never got picked up is a booking failure with its
own sweep, not a delivery problem.

Custody must accept any movement code, not just AC, because carriers signal
it inconsistently. CHR emits LOAD PICKED UP / CARRIER ARRIVED (AC) and
ShipEngine LTL usually opens with an AC scan (R&L, XPO) — but not always:
DE796341's five Saia scans are all IT, as are DE790921's eleven R&L ones.
An AC-only guard would silently exempt those from ever being flagged.

Shared with the parcel side (no_pickup_scan?), where it means the same
thing from the other direction: everything here is "the carrier physically
has it". The two codes deliberately left out are the two that mean it
doesn't — NY (label created / awaiting pickup) and UN (unknown).

(['AC'] + ShipmentEvent::MOVEMENT_STATUS_CODES).freeze
NO_PICKUP_SCAN_GRACE =

How long after we register a label with ShipEngine a parcel may sit
without a single custody scan before it counts as a problem.

This is the early-warning criterion: it fires while the parcel is still
young, off the label alone, where overdue? cannot — a parcel the carrier
never scanned gets no estimated_delivery_date from ShipEngine, and
overdue? needs one to have a date to be past. All 19 currently-open
no-scan parcels measured on 2026-08-05 had a NULL estimated_delivery_date,
so today they are invisible to every other criterion no matter how long
they sit.

Measured from tracking_registered_at (stamped by
ShipmentTrackingRegistrationWorker when the number is handed to
ShipEngine) rather than date_shipped, which is a DATE: from midnight the
first-scan p90 is 21h, so a date-based grace is mostly measuring what time
of day the warehouse prints labels. Against the real timestamp, over 2,598
parcels shipped in the 60 days to 2026-08-05, the first custody scan lands
at a 4.7h p90 and a 6.0h p95, with 98.96% inside 24h (99.19% at 36h,
99.35% at 48h).

Counted in WORKING time, not wall clock (config/initializers/ working_hours.rb: Mon–Fri 08:30–17:30 America/Chicago). 18 working hours
is two working days, which lands ~48 wall-clock hours after a weekday
label — the "extra day" — while a Friday-afternoon label doesn't come due
until Tuesday instead of alerting into an empty Saturday. One
working_time_between call covers both the afternoon-label and the
weekend case; the earlier wall-clock version handled neither, and Friday
labels measurably ran worse than midweek (2.7% past 24h vs ~0.9%).

The global WorkingHours config carries no holidays — only the per-company
config does (Company#with_working_hours_config), and reaching a company
from a shipment is three associations for one day a year. A holiday label
simply comes due a day early.

18.hours
EXPRESS_NO_PICKUP_SCAN_GRACE =

Express gets half that — one working day. An overnight parcel that has not
been scanned into custody by the end of the next working day has already
missed the commitment, so waiting the full ground grace would mean the tab
goes red a day after the customer has noticed. Also in working time, so a
Friday express label is asked about on Monday rather than on Saturday.

9.hours
DELIVERY_LEVEL_REASONS =

Reasons that describe the whole delivery rather than one parcel, and so
collapse to one tab row and one alert per delivery (collapse_key,
#cache_key). A consignment nobody picked up is one failure at the dock
however many boxes were on the pallet — see collapse_key.

%i[no_pickup_scan].freeze
ORIGIN_COUNTRY_ISO3 =

Origin countries this sweep speaks for, matched on the DELIVERY's origin
address — the dock the parcel left from, not where it is going. A US→FR
export still leaves our Illinois dock on a FedEx pickup, so it stays in
scope; a Netherlands→France DPD move never touches that network at all.

Every threshold in this file was measured on North American carriers, so
applying them to a European move is asserting something we never checked.
DE795826 is the case: five DPD boxes NLD→FRA, the first candidate the
no_pickup_scan criterion ever produced, and its silence says nothing we
can read. The whole EU population is 5 parcels in 30 days, so this costs
no North American coverage.

%w[USA CAN].freeze
FIXED_SEVERITY =

Reasons whose severity is a property of the reason itself, not of how late
the parcel is. Everything else is graded by lateness — see severity.

  • +exception+ is a carrier telling us it could not deliver: refused,
    returned to sender, damaged, address unknown. That is a failure on its
    own terms, and a returned-to-sender parcel is not "one day late".
  • +pickup_in_lieu+ arrived at a locker or access point instead of the
    address. Critical because it is the one failure the CUSTOMER has to
    resolve and cannot resolve without being told: they are expecting a
    parcel at their door, the carrier is holding it somewhere else on a
    clock, and when that clock expires it goes back to the shipper and the
    order fails outright. Being on time is no comfort to someone who does
    not know where their package is.

Marked critical "at least for now" (2026-08-06) — deliberately ahead of
the evidence, because the cost of being wrong is a needless red row while
the cost of being right and quiet is a returned order. Revisit once the
digest shows how many of these ops actually has to intervene on.

{ exception: :critical, pickup_in_lieu: :critical }.freeze
CRITICAL_DAYS_LATE =

Days past the delivery promise at which a late parcel turns from yellow to
red, per service tier.

Two tiers because the same slip means different things. Missing a ground
commitment by a day is inside normal carrier variance; missing an
OVERNIGHT commitment by a day means the customer has already not got the
thing they paid extra for, and no amount of catching up fixes it.

Express is therefore red the moment it is late at all — including a
missing pickup scan, which for an overnight has already burned the whole
commitment (see severity).

{ express: 1, ground: 3 }.freeze
ASSUMED_TRANSIT_DAYS =

Stand-in transit time for a parcel so dark the carrier never gave us an
estimated delivery date — the no_pickup_scan population, which is
exactly the set with no promise to be measured against.

4 days sits between the p50 (2.0) and p90 (6.0) of what carriers actually
promise from the label, measured over 2,095 US/CA parcels that DID get an
estimate in the 60 days to 2026-08-06. Combined with the ground threshold
above it puts a dark ground parcel in the red at 7 days of silence, which
is independently where it has outlived 95% of its delivered peers
(label-to-delivered p95 = 6.9 days over 2,457 parcels).

4
ALERT_CACHE_TTL =

Constant.

7.days
MAX_ALERTS_PER_RUN =

Blast-radius cap per hourly run: a carrier-wide incident (every UPS
parcel in the 30-day lookback throwing EX scans at once) must not
enqueue an unbounded flood of escalation emails. Uncapped candidates
simply wait — the next hourly run alerts the next batch, and the
7-day claim keeps already-alerted shipments from re-sending.

100
EXCEPTION_TRACKING_STATES =

Roll-up tracking_state fallbacks per reason, for shipments whose scans
predate the shipment_events projection (or arrived via a path that only
updates the Shipment roll-up columns).

%w[exception delivery_attempt].freeze
FEDEX_MPS_TRACKING_FORMAT =

FedEx tracking numbers are 12 digits. Purolator's domestic-CA numbers
share that shape but start 520, so they're excluded — Purolator is not
a multi-piece-consignment carrier for our purposes and must not inherit
a sibling's delivered scan.

/\A(?!520)\d{12}\z/
CANPAR_TRACKING_FORMAT =

Canpar's 22-character barcode: 'D' + 8-digit account + zero padding +
shipment sequence + 3-digit piece suffix ('001' = the lead box, the
only piece Canpar ever emits scans for — 0 of 54 non-lead boxes have
ever scanned).

/\AD\d{21}\z/

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.badge_reason(shipment, events: nil) ⇒ Symbol?

Badge-path variant of problem_reason for a single already-loaded
shipment outside the sweep query (delivery rows / show page):
applies the same eligibility guards as candidates in Ruby, then
classifies. Returns nil for ineligible shipments (dismissed,
delivered roll-up, outside the lookback, voided, no tracking).

Parameters:

  • shipment (Shipment)
  • events (Array<ShipmentEvent>, nil) (defaults to: nil) —

    pre-loaded scans (list views)

Returns:

  • (Symbol, nil) 


381
382
383
384
385
 
# File 'app/workers/problematic_delivery_sweep.rb', line 381

def badge_reason(shipment, events: nil)
  return nil unless badge_eligible?(shipment)

  problem_reason(shipment, events: events)
end

.candidates(store: nil) ⇒ ActiveRecord::Relation<Shipment>

Recently shipped, completed, tracked parcel shipments not yet rolled
up as delivered and not manually dismissed. Includes
'delivered_to_collection_location' (the pickup-in-lieu signal) —
only a plain 'delivered' roll-up exits the candidate set.

Parameters:

  • store (Store, nil) (defaults to: nil) —

    restrict to one warehouse's deliveries

Returns:



246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
 
# File 'app/workers/problematic_delivery_sweep.rb', line 246

def candidates(store: nil)
  scope = Shipment.top_level.completed
                  .where.not(carrier: 'SpeedeeDelivery')
                  .where.not(delivery_id: nil)
                  .where.not(tracking_number: [nil, ''])
                  .where.not(tracking_state: 'delivered')
                  .where(problematic_dismissed_at: nil)
                  .where(date_shipped: LOOKBACK_WINDOW.ago.to_date..)
                  .joins(delivery: :origin_address)
                  .where(addresses: { country_iso3: ORIGIN_COUNTRY_ISO3 })
                  # shipping_option and order back {severity}'s service tier
                  # and requested_deliver_by lookups, one per row otherwise.
                  .includes(:shipment_events, :order,
                            delivery: %i[rma_for_return freight_events shipping_option order])
  scope = scope.where(delivery_id: Delivery.by_store_id(store.id).select(:id)) if store
  scope
end

.days_late(shipment) ⇒ Integer

Whole days past the delivery promise; 0 or negative when not yet late.

Parameters:

Returns:

  • (Integer) 


330
331
332
333
334
335
 
# File 'app/workers/problematic_delivery_sweep.rb', line 330

def days_late(shipment)
  promise = promise_date(shipment)
  return 0 if promise.nil?

  (Date.current - promise).to_i
end

.express?(shipment) ⇒ Boolean

Overnight / expedited service, where a one-day slip is already a failed
promise. Read from the delivery's shipping option — present on 100% of
candidates — via ShippingOption#express?, the row-level twin of the
scope Order::ExpressHoldAlerting#express_shipping? matches on.

Parameters:

Returns:

  • (Boolean) 


368
369
370
 
# File 'app/workers/problematic_delivery_sweep.rb', line 368

def express?(shipment)
  shipment.delivery&.shipping_option&.express? || false
end

.flagged(store: nil) ⇒ Array<Array(Shipment, Symbol)>

Every currently problematic shipment with its reason, one row per
physical parcel: sibling shipment rows sharing a tracking number on
the same delivery (FBA multi-box transfers stamp one 1Z number on
every box row — DE796327) collapse to the lowest-id row.

The +order(:id)+ is what makes "collapse to the first" mean anything:
candidates carries no ORDER BY, so which sibling survived the +uniq+
was whatever order PostgreSQL happened to return — the warehouse tab
could show a different box row for the same parcel between page loads.
Surfaced by the collapse test failing in CI on 2026-07-28 and passing on
a re-run of the identical commit. Ordering lives HERE, not on
candidates, because +uniq+ is the only caller that needs it; #perform
batches candidates through +find_each+, which imposes its own
primary-key batch order regardless.

Alerting is unaffected either way: #cache_key is keyed on tracking_number
and reason rather than shipment id, so siblings claim one alert between
them PER REASON no matter which row wins here. (Siblings usually classify
alike — #problem_reason reads scans by tracking_number, which they share —
but the roll-up fallbacks read per-row columns, so two boxes CAN land on
different reasons and then alert separately. That is deliberate: two
distinct problems on one parcel are two things for ops to work.)

Parameters:

  • store (Store, nil) (defaults to: nil)

Returns:



289
290
291
292
293
294
 
# File 'app/workers/problematic_delivery_sweep.rb', line 289

def flagged(store: nil)
  candidates(store:).order(:id).filter_map do |shipment|
    reason = problem_reason(shipment)
    [shipment, reason] if reason
  end.uniq { |shipment, reason| collapse_key(shipment, reason) }
end

.freight?(delivery) ⇒ Boolean

Parameters:

Returns:

  • (Boolean) 


457
458
459
 
# File 'app/workers/problematic_delivery_sweep.rb', line 457

def freight?(delivery)
  delivery&.ltl_freight.present?
end

.problem_reason(shipment, events: nil) ⇒ Symbol?

First matching reason wins, most severe first.

Scans are read by tracking_number, NOT via the shipment_events
association: one physical parcel can back several shipment rows
(Override re-ships, US→CA stock-transfer consolidations put the same
tracking number on 3 deliveries — DE793883/DE793885/DE794035) and
the webhook processor attaches each scan to only one of them. A
delivered scan on a sibling row must clear this shipment too. The
returned-to-sender trap (the final DE being the parcel arriving back
at origin, DE792884) is handled inside exception? instead.

Parameters:

  • shipment (Shipment)
  • events (Array<ShipmentEvent>, nil) (defaults to: nil) —

    pre-loaded scans for the
    tracking number (list views); nil queries by tracking_number

Returns:

  • (Symbol, nil) —

    :exception, :pickup_in_lieu, :no_pickup_scan,
    :overdue_no_delivery



403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
 
# File 'app/workers/problematic_delivery_sweep.rb', line 403

def problem_reason(shipment, events: nil)
  events = (events || scans_for(shipment)).sort_by(&:occurred_at)
  # `||` short-circuits: the consignment lookup only runs for a piece with
  # no delivered scan of its own, which is the only case it can change.
  last_de_at = events.rfind { |e| e.status_code == 'DE' }&.occurred_at ||
               consignment_delivered_at(shipment) ||
               goods_receipted_at(shipment)

  return :exception if exception?(shipment, events, last_de_at)
  return :pickup_in_lieu if pickup_in_lieu?(shipment, events, last_de_at)
  # Before :overdue_no_delivery — both can match a parcel that never
  # scanned, and "the carrier never took it" is the sharper diagnosis.
  return :no_pickup_scan if no_pickup_scan?(shipment, events, last_de_at)
  return :overdue_no_delivery if overdue?(shipment, events, last_de_at)

  nil
end

.promise_date(shipment) ⇒ Date?

The date this parcel was supposed to arrive.

The EARLIEST commitment on file wins, because the first promise we break
is the one that matters — the carrier's estimate, or the customer's own
requested_deliver_by when they needed it sooner. In practice they
rarely disagree in that direction (7 of 1,470 shipments over 30 days),
so this is mostly "the carrier's estimate, unless the customer needed it
earlier".

Falls back to ASSUMED_TRANSIT_DAYS past the label for parcels with no
promise at all — a carrier that never scanned the parcel never gave us
an estimate either, so the dark population would otherwise be ungradable
and sit yellow forever.

Parameters:

Returns:

  • (Date, nil) 


353
354
355
356
357
358
359
 
# File 'app/workers/problematic_delivery_sweep.rb', line 353

def promise_date(shipment)
  order = shipment.order || shipment.delivery&.order
  promises = [shipment.estimated_delivery_date, order&.requested_deliver_by].compact
  return promises.min if promises.any?

  shipment.tracking_registered_at&.to_date&.+(ASSUMED_TRANSIT_DAYS)
end

.scans_for(shipment) ⇒ Array<ShipmentEvent, FreightEvent>

The scans that describe this shipment's journey.

LTL freight is a DELIVERY-level move, not a parcel-level one: one truck
carries the whole consignment, so per-shipment tracking numbers describe
nothing the carrier reports on. (Measured: 93 of 98 LTL deliveries over
180 days have exactly one shipment row anyway — 3 have two, 2 have
three.) So freight reads at the delivery level, from BOTH freight
sources, unioned rather than branched on carrier:

A delivery only ever has one of the two, so CHR is checked first and the
PRO lookup is the fallback — which also keeps the warehouse dashboard
query-free for Freightquote rows, whose freight_events are preloaded by
Crm::WarehousesController#preload_delivery_tracking_events!. Only
ShipEngine LTL pays a query, and there were 9 such deliveries in 120 days.

Parameters:

Returns:



444
445
446
447
448
449
450
451
452
453
 
# File 'app/workers/problematic_delivery_sweep.rb', line 444

def scans_for(shipment)
  delivery = shipment.delivery
  return ShipmentEvent.for_tracking_number(shipment.tracking_number).to_a unless freight?(delivery)

  chr = delivery.freight_events.to_a
  return chr if chr.any?

  pro = delivery.ltl_pro_number
  pro.present? ? ShipmentEvent.for_tracking_number(pro).to_a : []
end

.severity(shipment, reason) ⇒ Symbol

How loudly this row should read: +:critical+ (red) or +:warning+
(yellow). Drives the warehouse tab's badge and the digest's ordering.

FIXED_SEVERITY reasons answer on their own — an exception is a failure
whatever the calendar says. The two lateness reasons (+no_pickup_scan+,
+overdue_no_delivery+) are graded on HOW late against WHICH promise, per
CRITICAL_DAYS_LATE: a ground parcel is yellow for its first two days
late and red from the third, while an overnight is red as soon as it
slips at all.

An express parcel with no pickup scan is red outright. There is no such
thing as an overnight that has not been picked up but is still on time,
and EXPRESS_NO_PICKUP_SCAN_GRACE means we only ask once a working day
has passed.

Parameters:

  • shipment (Shipment)
  • reason (Symbol, String)

Returns:

  • (Symbol) —

    :critical or :warning



314
315
316
317
318
319
320
321
322
323
324
 
# File 'app/workers/problematic_delivery_sweep.rb', line 314

def severity(shipment, reason)
  reason = reason.to_sym
  fixed = FIXED_SEVERITY[reason]
  return fixed if fixed

  express = express?(shipment)
  return :critical if reason == :no_pickup_scan && express

  threshold = CRITICAL_DAYS_LATE.fetch(express ? :express : :ground)
  days_late(shipment) >= threshold ? :critical : :warning
end

Instance Method Details

#perform ⇒ Object

Runs the job.

Returns:

  • (Object) —

    the result



796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
 
# File 'app/workers/problematic_delivery_sweep.rb', line 796

def perform
  alerted = 0
  self.class.candidates.find_each do |shipment|
    reason = self.class.problem_reason(shipment)
    next unless reason

    if alerted >= MAX_ALERTS_PER_RUN
      Rails.logger.warn "[ProblematicDeliverySweep] per-run alert cap (#{MAX_ALERTS_PER_RUN}) " \
                        'reached — remaining candidates deferred to the next hourly run'
      break
    end

    next unless claim_alert!(shipment, reason)

    alerted += 1
    alert!(shipment, reason)
  end
end