Class: StaleTranscriptionRecoveryWorker

Inherits:
Object
  • Object
show all
Includes:
Sidekiq::Job
Defined in:
app/workers/stale_transcription_recovery_worker.rb

Overview

Recovers stale push-pull webhooks that never received a callback.

This worker runs every 2 hours and:

  1. Finds WebhookLog entries stuck in "pending" state for > 1 hour (awaiting callback)
  2. Checks the provider directly by job id — AssemblyAI (transcriptions) and
    Oxylabs (retailer price checks) are both push-pull, so a lost callback is
    still pullable by job id (and pulling costs no new Oxylabs scrape budget)
  3. If completed, transitions to ready and processes
  4. If still processing at the provider, leaves alone
  5. If failed/faulted, marks as exception
  6. Also processes any WebhookLog entries stuck in "ready" or "retry" state

This is a safety net - webhooks should handle 99%+ of completions.
This worker catches edge cases like:

  • Webhook delivery failures
  • Network issues during webhook POST
  • Server restarts during processing
  • Worker processing failures

Schedule: Every 2 hours via sidekiq-scheduler

Constant Summary collapse

STALE_THRESHOLD =

How long to wait before considering a webhook stale (no callback received)

1.hour
MAX_AGE =

Maximum age to check (don't process ancient stuck records)

48.hours
BATCH_LIMIT =

Limit per run to avoid overwhelming the system

50
PROCESSING_STUCK_AFTER =

How long a row may claim processing before we stop believing it. Measured
from transcription_started_at, NOT created_at: the importers backdate
created_at to the time of the CALL, and the bulk backfill transcribes calls
that are years old, so a created_at predicate would call every in-flight
backfill record stale the instant it started.

6.hours
RECLAIM_LIMIT =

Per-run cap on reclaims. The population is 423 records today, but the cap is
what keeps a future incident from turning one run into a mass state change.

100

Instance Method Summary collapse

Instance Method Details

#perform ⇒ void

This method returns an undefined value.

Sweeps stale pending webhooks, processes any ready/retry logs, then
reclaims wedged processing call records.



53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
 
# File 'app/workers/stale_transcription_recovery_worker.rb', line 53

def perform
  Rails.logger.info '[StaleTranscriptionRecoveryWorker] Starting stale transcription recovery'

  stats = { recovered: 0, failed: 0, still_processing: 0, ready_processed: 0, reclaimed: 0 }

  # First, check stale pending entries (jobs that never got a callback)
  check_stale_pending_webhooks(stats)

  # Then, process any ready/retry webhook logs that need processing
  process_ready_webhook_logs(stats)

  # Finally, un-wedge call records left claiming to be `processing`
  reclaim_wedged_call_records(stats)

  Rails.logger.info "[StaleTranscriptionRecoveryWorker] Completed: #{stats}"
end