Skip to content
Heatwave Docs

Freightquote / CH Robinson Events Pipeline

Status: in progress on branch rb_fqchr_events (PR #844). This doc describes the design as it will land when the PR merges.

Why

Section titled “Why”

On 2026-05-19 delivery DE787078 was booked through Freightquote (FQ — CH Robinson’s LTL marketplace), routed to Shipby.com / Unis Transportation (SCAC UTPA). Shipby accepted the load, then verbally told CHR they couldn’t accommodate the customer’s inside-delivery requirement and ghosted. The carrier never picked up. We learned about it ~27 hours later from a customer phone call. Juan then sat on hold with FQ for 30 minutes to manually void and rebook through R+L Carriers.

CHR’s events API did eventually emit LOAD CANCELLED and ORDER CANCELED for this load — but only ~20 hours after Juan’s manual void, triggered by the rep cleaning up the order in their internal system after an email exchange. We weren’t subscribed to events at all, so we missed both the cancellation signal and (more importantly) the absence-of-pickup signal.

Forensic write-up: see chat thread on 2026-05-20 and doc/tasks/202605201129_FREIGHTQUOTE_VOID_CONFIRMATION.md.

What this pipeline does

Section titled “What this pipeline does”

Receives CHR Navisphere events (LOAD CREATED, LOAD BOOKED, LOAD CANCELLED, ORDER REJECTED, APPOINTMENT UPDATED, PRO NUMBER ADDED, LOAD PICKED UP, LOAD DELIVERED, etc.) via webhook callback, persists each one to a queryable freight_events table, and fires side effects on the failure-signal types (rejection/cancellation events page ops via AppSignal).

Two scheduled workers backstop the webhook path: MissedFreightPickupSweep (hourly) pages ops when a pickupByDate passes without a LOAD PICKED UP event — the actual DE787078 catch. FreightquoteVoidConfirmationWorker (one-shot, scheduled 1h after a void) pages ops when CHR doesn’t echo our DELETE with a LOAD CANCELLED / ORDER CANCELED event. See the Backstop signals section below.

Architecture

Section titled “Architecture”
┌─────────────────────────────┐
│  CH Robinson Navisphere     │
│  (Events Callback product)  │
└────────────┬────────────────┘
             │ POST application/json
             │ Basic auth
             ▼
┌─────────────────────────────────────────┐
│ Webhooks::V1::FreightquoteController    │
│ - verify Basic auth                     │
│ - parse JSON                            │
│ - resolve delivery_id from              │
│   customerReferenceNumber               │
│ - WebhookLog.ingest! with synthesised   │
│   external_id (idempotency_key)         │
│ - WebhookProcessorWorker.perform_async  │
│ - return 200                            │
└────────────┬────────────────────────────┘
             │ Sidekiq
             ▼
┌─────────────────────────────────────────┐
│ WebhookProcessors::FreightquoteProcessor│
│ - FreightEvent.parse_payload            │
│ - FreightEvent.find_by(idempotency_key) │
│   → return 'duplicate' if exists        │
│ - FreightEvent.create!                  │
│ - dispatch_side_effects                 │
└────────────┬────────────────────────────┘
             │
             ├──→ ORDER REJECTED / LOAD CANCELLED / ORDER CANCELED
             │    → ErrorReporting.warning (AppSignal background_warning)
             │
             ├──→ LOAD PICKED UP  (future: confirm in-transit milestone)
             ├──→ LOAD DELIVERED  (future: trigger Delivery#shipped! if pending)
             └──→ everything else: persist only

Two data stores, by design:

  • webhook_logs.data (JSONB) — the raw audit trail; state machine for retries; webhook_logs CRM screen lists every incoming event regardless of provider. Source of truth for “did CHR send us this?”
  • freight_events — typed/indexed projection. One row per Navisphere event with event_type, event_time, load_number, order_number, carrier_scac, pro_number, etc. as columns. Source of truth for “what is the current state of delivery X according to CHR?” Fast to query for the pickup-window sweep and any future operational dashboard.

CHR doesn’t emit a stable event ID, so we synthesise an idempotency_key from event_type | event_time(iso8601μs) | load_number | order_number. Unique index enforces dedupe; the processor returns 'duplicate' without re-firing side effects when CHR retries a webhook.

CHR setup (one-time per environment)

Section titled “CHR setup (one-time per environment)”

This part is non-self-serve and requires coordination with our CHR account team (Travis Julo / Andrew Erwin / Micah Tedford).

1. Pick auth + URL

Section titled “1. Pick auth + URL”

CHR’s Events Callback product supports Basic Auth, OAuth 1.0, OAuth 2.0, API key, or No Auth (per the Navisphere swagger at ~/Downloads/navisphere_api_b2b_v1_doc_swagger.yaml, line 3378). We use Basic Auth to match the existing Webhooks::V1::ShipengineController pattern.

EnvironmentWebhook URLAuth
Dev (tunnelled)https://api.warmlyyours.dev/webhooks/v1/freightquoteper-dev creds, or open
Staginghttps://api.warmlyyours.me/webhooks/v1/freightquoteTBD creds
Productionhttps://api.warmlyyours.com/webhooks/v1/freightquoteprod creds

2. Configure credentials

Section titled “2. Configure credentials”

Heatwave reads Basic auth credentials from Heatwave::Configuration:

Heatwave::Configuration.fetch(:freightquote, :webhook_username)
Heatwave::Configuration.fetch(:freightquote, :webhook_password)

Production security requirements — both are enforced in code:

  • HTTPS-only at the edge. The endpoint MUST be reachable only over HTTPS at the load balancer / reverse proxy; no HTTP fallback. Basic Auth over HTTP would expose the credential on every request — its threat model assumes a TLS-terminated channel. Before pointing CHR at the prod URL, confirm curl http://api.warmlyyours.com/webhooks/v1/freightquote -X POST returns a redirect or refusal, not a 401/200.
  • Both credentials configured. webhook_username AND webhook_password must be present in the encrypted credentials for every non-dev environment. The controller fails closed with 401 in Rails.env.production? when either is blank (verify_basic_auth!). In dev/test the open fallback is preserved for bring-up convenience — same pattern as Webhooks::V1::ShipengineController.

Add both to the encrypted credentials file for each environment before asking CHR to point at us.

3. Pick the event-type subscription

Section titled “3. Pick the event-type subscription”

Per the swagger (line 3477+), CHR lets us pick which event types they push. For LTL via FQ, the useful set is:

Order-scope (5): ORDER CREATED, ORDER REJECTED, ORDER UPDATED, ORDER CANCELED, ORDER COMPLETED Load-scope (4): LOAD CREATED, LOAD BOOKED, LOAD CANCELLED, PRO NUMBER ADDED Stop-scope (4): CARRIER ARRIVED, CARRIER DEPARTED, LOAD PICKED UP, LOAD DELIVERED Appointment (1): APPOINTMENT UPDATED In-transit (2): IN TRANSIT, IN TRANSIT TO ORIGIN Tracking (1): CHR TRACKING NUMBER PUBLISHED

Skip: package-scope (small-parcel), global-forwarding shipment/container/customs (we don’t ship those lanes via FQ).

4. Ask Travis to configure

Section titled “4. Ask Travis to configure”

Send Travis a short email with the URL, auth username, auth password (out-of-band — don’t email passwords), and the event-type list. CHR’s side is point-and-click in their admin; usually live within a business day.

5. Verify with a single live event

Section titled “5. Verify with a single live event”

Once CHR confirms the subscription is active, ask them to send a test event, or wait for the next real FQ booking on our side. The controller logs every incoming request; check webhook_logs (filter provider: 'freightquote') for the row. If it landed, the pipeline is alive.

Event types we handle (this PR)

Section titled “Event types we handle (this PR)”
Event typeWhat it meansSide effect today
ORDER REJECTEDCHR refused the booking, won’t continueAppSignal warning
LOAD CANCELLEDLoad cancelled (carrier dropped or we voided)AppSignal warning
ORDER CANCELEDOrder fully unwoundAppSignal warning
Everything else (LOAD CREATED, LOAD BOOKED, LOAD PICKED UP, LOAD DELIVERED, APPOINTMENT UPDATED, ORDER UPDATED, IN TRANSIT, PRO NUMBER ADDED, CARRIER ARRIVED/DEPARTED, CHR TRACKING NUMBER PUBLISHED, ORDER CREATED, ORDER COMPLETED)Lifecycle / milestone signalPersist to freight_events, no other side effect

Side effects we deliberately do not do (yet):

  • Auto-transition Delivery state back to pending_ship_labels on a rejection. Delivery#cancel_shipments is gated on shipments.completed.any? being false, which never holds for our post-label rejection cases. Auto-voiding via WyShipping.void_delivery from inside an event handler is risky; for now we rely on ops working the AppSignal alert and voiding from the CRM.
  • Auto-transition to shipped on LOAD PICKED UP. Same reasoning — coordination with the warehouse ship_confirm workflow needed first.

Backstop signals (no CHR webhook required)

Section titled “Backstop signals (no CHR webhook required)”

Two scheduled workers fill the gap when CHR’s webhook either doesn’t emit or arrives too late:

MissedFreightPickupSweep — pickup-window safety net

Section titled “MissedFreightPickupSweep — pickup-window safety net”

Runs hourly via sidekiq-scheduler (config/sidekiq_{production,staging}_schedule.yml). For each LOAD BOOKED FreightEvent on an active delivery (pending_ship_confirm / pending_carrier_confirm), checks whether a LOAD PICKED UP event for the same loadNumber exists. If not AND the booked pickupByDate is past 1 business day (America/Chicago), pages ops via ErrorReporting.warning.

This is the actual DE787078 catch. When Shipby verbally told CHR they couldn’t do the inside delivery, no electronic event fired — but the absence of LOAD PICKED UP after Friday’s pickup window would have alerted us Monday morning instead of Tuesday afternoon.

Re-alert suppression is via Rails.cache (Redis) with a 7-day TTL keyed by delivery_id, so the same delivery doesn’t re-page every hour while ops works the issue.

FreightquoteVoidConfirmationWorker — void-DELETE confirmation

Section titled “FreightquoteVoidConfirmationWorker — void-DELETE confirmation”

Scheduled by Delivery#void_shipments at +1 hour after a successful FQ void. Checks two sources for a LOAD CANCELLED or ORDER CANCELED echo of our void, in order:

  1. Fast path — local FreightEvent rows populated by Webhooks::V1::FreightquoteController when CHR pushes a callback. Cheap DB query; the canonical source once CHR’s Customer Integration Team has us configured.
  2. Fallback — direct GET /v2/events?orderNumber=… against CHR via WyShipping.freightquote_cancel_event_present?. Uses the same OAuth bearer we mint for rate/label/void calls. Does NOT require the webhook subscription to be configured — so the worker is operationally useful before Travis has us set up, and acts as defense-in-depth against missed webhooks afterward.

Alerts via ErrorReporting.warning only when both sources say “no cancel event in the window” (the void timestamp ± 5 min). A /v2/events poll error (CHR auth blip, network hiccup) is treated as “not confirmed” — false-positive is safer than missed-cancel.

Non-blocking — same notify-and-move-on pattern as the existing Canadapost / Purolator manual-void email fallback. Order and delivery flows do NOT wait on this confirmation.

Operational handling

Section titled “Operational handling”

When ops sees a Freightquote carrier rejection AppSignal warning:

  1. Click into the AppSignal incident. Context includes delivery_id, delivery_state, event_type, chr_order_number, chr_load_number, carrier_scac, freight_event_id, webhook_log_id.
  2. Open the delivery in the CRM at /orders/<order_id>?tab=shipping. The shipping tab shows the FQ identifiers (BOL, load number, PRO) and the shipping_api_log JSONB which now records each label/void/rate/events call to CHR.
  3. Did Juan / the warehouse already void? Check shipping_api_log for a kind: 'void' entry. If yes, the cancel event was the expected echo and you can dismiss.
  4. Otherwise — call CHR. The carrier dropped the load on their side; we need a new carrier or a refund. Travis Julo / our CHR rep team can replan.
  5. Manually void in our system. Go to the delivery’s shipping tab and click void (or use the CRM action), then re-rate and rebook with a different carrier.

Testing the pipeline

Section titled “Testing the pipeline”

Three ways, in increasing realism:

A) Unit tests

Section titled “A) Unit tests”

Already wired. Run:

Terminal window
PARALLEL_WORKERS=1 mise exec -- bin/rails test \
  test/models/freight_event_test.rb \
  test/services/webhook_processors/freightquote_processor_test.rb \
  test/controllers/webhooks/v1/freightquote_controller_test.rb

Covers payload parsing across event variants, idempotency on replay, side-effect dispatch firing only for rejection events, ingest + auth + queueing.

B) Local replay with the canned fixtures

Section titled “B) Local replay with the canned fixtures”

test/fixtures/files/freightquote/ holds 9 sample payloads — 7 real (the entire DE787078 event sequence pulled live from CHR on 2026-05-20) plus 2 synthetic (ORDER REJECTED, LOAD PICKED UP) for happy/sad paths CHR didn’t emit for that specific delivery. See the fixtures README for the catalog.

Start the dev server, then:

Terminal window
# List what's available
script/freightquote_event_replay.sh --list


# Replay LOAD CANCELLED against local dev, mapped to a real dev delivery
script/freightquote_event_replay.sh load_cancelled --delivery-id 42


# Replay against staging (Basic auth read from env)
script/freightquote_event_replay.sh order_rejected \
  --url https://api.warmlyyours.dev/webhooks/v1/freightquote \
  --auth "$FQ_WEBHOOK_USER:$FQ_WEBHOOK_PASS"

A successful round-trip:

  1. Controller returns 200.
  2. WebhookLog row appears with provider: 'freightquote', category matching the lowercased event type, external_id set to the synthesised idempotency key.
  3. WebhookProcessorWorker runs (visible in Sidekiq UI).
  4. FreightEvent row appears, linked to the delivery if customerReferenceNumber resolved.
  5. For rejection types: an AppSignal warning appears in the background_warning namespace.

To verify in a Rails console:

FreightEvent.order(created_at: :desc).limit(5)
WebhookLog.where(provider: 'freightquote').order(created_at: :desc).limit(5)

C) Direct processor invocation (Rails / prod console)

Section titled “C) Direct processor invocation (Rails / prod console)”

Skips HTTP entirely — useful if you want to test the parser/side-effect logic without standing up a server, or to replay a real CHR payload pulled from Shipping::Freightquote#check_events_for_load_number:

# Load a fixture (or paste a real payload from CHR's events API)
payload = JSON.parse(File.read('test/fixtures/files/freightquote/load_cancelled.json'))


# Optionally rewrite customerReferenceNumber to a real dev delivery
payload['customerReferenceNumber'] = '42'


# Synthesise a WebhookLog and run the processor
webhook_log = WebhookLog.create!(
  provider: 'freightquote',
  category: payload['event']['eventType'].downcase.tr(' ', '_'),
  data: payload,
  external_id: FreightEvent.idempotency_key_for(
    event_type: payload['event']['eventType'],
    event_time: payload['eventTime'],
    load_number: payload.dig('event', 'loadNumber'),
    order_number: payload.dig('event', 'orderDetails', 0, 'orderNumber') ||
                  payload.dig('event', 'orderDetail', 0, 'orderNumber')
  ),
  state: 'ready'
)
WebhookProcessors::FreightquoteProcessor.call(webhook_log)

D) End-to-end with CHR

Section titled “D) End-to-end with CHR”

Once Travis has the subscription active, the next real LTL booking is your end-to-end test. The booking + LOAD CREATED + LOAD BOOKED + APPOINTMENT UPDATED + ORDER CREATED + ORDER UPDATED sequence will flow within ~30 seconds. Watch the webhook_logs CRM screen for them to land.

To force a LOAD CANCELLED without waiting: void any active FQ delivery in the CRM. CHR should echo the cancel within minutes (or hours, per DE787078’s experience).

Known limitations

Section titled “Known limitations”

  1. CHR’s signal lag is not real-time for human-mediated cases. When Shipby ghosts because they can’t perform the service, no electronic event fires. The cancel event only arrives after our rep manually cleans up. MissedFreightPickupSweep (above) closes this — it doesn’t depend on CHR emitting anything; the absence of LOAD PICKED UP past the booked pickup window is the signal.

  2. WebhookLog dedupe gap. WebhookLog.ingest! only short-circuits when an existing entry is in processed/processing/pending/retry/exception state. ready-state entries fall through and create a duplicate row. Affects every provider, not just FQ. The data-integrity guarantee for FQ events is on freight_events.idempotency_key (unique index) — the processor returns 'duplicate' cleanly.

  3. No webhook-side signature verification. CHR’s webhook auth is Basic only (no HMAC signing in their swagger). If the webhook URL and Basic Auth credentials both leak, an attacker can:

    • Inject false ORDER REJECTED / LOAD CANCELLED / ORDER CANCELED events → spurious AppSignal pages (ops-time DoS).
    • Insert fraudulent FreightEvent rows → poison operational dashboards and the webhook_logs audit trail.
    • Forge a LOAD PICKED UP for a load that was never picked up → trip MissedFreightPickupSweep’s Rails.cache alert suppression for that delivery_id and mask a real pickup failure for up to 7 days.

    Mitigations (operational, not in-code):

    • Treat the webhook URL + Basic Auth pair as a high-sensitivity secret. Rotate immediately on any suspicion of compromise — the controller picks up new credentials on the next request without restart.
    • Monitor webhook_logs for spikes in provider: 'freightquote' ingest rate. CHR’s normal rate is a handful per active LTL booking per day; sustained higher rates are suspicious.
    • If CHR exposes a static source IP range for their Events Callback product (not currently documented in the swagger), restrict our endpoint at Cloudflare / the load balancer to that range. Worth asking Travis.

    Future enhancement: if CHR adds HMAC signing or mTLS to their Events Callback product, swap Basic Auth for that. Verification should use ActiveSupport::SecurityUtils.secure_compare to avoid timing oracles. Until then, the combination of HTTPS-only + secret credential + sensible Cloudflare WAF rules is acceptable.

  4. Pickup-window assumes America/Chicago timezone. CHR’s pickupByDate is a calendar date string with no timezone. The sweep treats it as Central Time (most of our LTL ships from Lake Zurich, IL). Canadian-origin LTL would be off by ~3 hours — acceptable since the deadline is “1 business day past pickup,” not minute-precise.

References

Section titled “References”