Class: DailyCallRecordTranscriptionWorker

Inherits:
Object
  • Object
show all
Includes:
Sidekiq::Job
Defined in:
app/workers/daily_call_record_transcription_worker.rb

Overview

Note:

the AssemblyAI backfill is OFF by default. The day-to-day coverage gap
is now closed by the cheap Gemini pipeline (CallRecordBulkTranscriptionWorker,
min_age 7d), which sweeps anything older than the recent window. Enable this
backfill only for a deliberate full-feature (diarization + LeMUR) re-run of a
chosen recent window — either by passing args to perform_async or by adding
them to the daily_call_record_transcription cron entry. It uses the expensive
AssemblyAI path, so it is intentionally not the default backlog tool.

Daily catch-up worker for call record transcriptions.

Primary transcription is queued immediately when call records are imported
via CallRecordSwitchvoxImporterFile. This worker serves as a safety net
to catch any records that were missed (e.g., queue issues, worker failures,
records created through other means).

Runs at 6 AM CT daily via sidekiq-cron.

The worker uses the ai_embeddings queue for controlled throughput
and to avoid impacting user-facing operations.

Cost estimate: ~$0.06/call (transcription + LeMUR + embedding)

Examples:

Full-feature AssemblyAI backfill of the last 30 days (100 calls)

DailyCallRecordTranscriptionWorker.perform_async(
  'backfill' => true, 'backfill_days' => 30, 'backfill_limit' => 100
)

Constant Summary collapse

BACKFILL_ENABLED =

Defaults for the optional AssemblyAI backfill (overridable per-run via args).

false
BACKFILL_LIMIT =

Constant.

500
BACKFILL_DAYS_LIMIT =

2 years = 730 days - older calls are not worth the AssemblyAI cost

730
RECONCILE_TOO_SHORT_DAYS =

How far back to give sub-floor pending calls their terminal too_short
verdict. They are below the transcription floor so CallRecord.transcription_eligible
permanently skips them; without this they sit in pending forever and inflate
the daily digest's "in flight" line. A rolling window backfills the recent stuck
ones and keeps each new day clean before the 8 AM digest.

30
RECONCILE_NO_AUDIO_DAYS =

How far back to give upload-less pending calls their terminal no_audio
verdict. CallRecord#created_at is BACKDATED to the call time by the importer,
and both CallRecord.transcription_eligible and CallRecord.transcription_too_short
INNER JOIN :upload — so a pending record that never got an audio file (a
voicemail CDR leg with no recording, a missed call) is unreachable by every
reconciler and sits in pending forever, inflating the digest's "in flight"
line. This is exactly the verdict CallRecordProcessing::TranscriptionService
assigns via skip_result(:no_audio), which these never reach. Mirrors
#reconcile_too_short: one bounded UPDATE over the recent window.

30
NO_AUDIO_GRACE =

Grace period before an upload-less pending call is declared no_audio.
Voicemail audio is attached out-of-band (VoicemailsMailbox / webhook) moments
after the record appears; this keeps a just-created record whose audio is still
arriving from being prematurely closed.

2.hours
RECONCILE_FILELESS_LIMIT =

Per-run cap on #reconcile_fileless_upload. Its siblings only ever see a
day's worth of strays, but this one drains the pre-fix backlog of records
stranded in processing (AppSignal #1982), so the first runs could otherwise
be one very large UPDATE. Matches BACKFILL_LIMIT; the remainder is picked up
by the next nightly pass, and the log says when the cap was hit.

500
REQUEUE_ELIGIBLE_DAYS =

How far back the catch-up re-queues stuck eligible (pending/error) calls.
The previous 24-hour window keyed on the BACKDATED created_at could never
catch an evening recording imported after that morning's catch-up run — by the
next run its call-time created_at is already >24h old, so it fell permanently
outside the net and was never transcribed (always the day's last recording).
A multi-day window closes that hole; re-queues are idempotent (eligible excludes
processing/completed, and the service skips already-transcribed records).

7

Instance Method Summary collapse

Instance Method Details

#perform(options = {}) ⇒ void

This method returns an undefined value.

Parameters:

  • options (Hash) (defaults to: {}) —

    run overrides; all optional

Options Hash (options):

  • :backfill (Boolean) —

    run the older-call backfill pass
    (defaults to BACKFILL_ENABLED)

  • :backfill_limit (Integer) —

    maximum calls the backfill pass
    enqueues (defaults to BACKFILL_LIMIT)

  • :backfill_days (Integer) —

    how far back the backfill pass
    reaches (defaults to BACKFILL_DAYS_LIMIT)



89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
 
# File 'app/workers/daily_call_record_transcription_worker.rb', line 89

def perform(options = {})
  backfill_enabled = options.fetch(:backfill, BACKFILL_ENABLED)
  backfill_limit   = (options[:backfill_limit] || BACKFILL_LIMIT).to_i
  backfill_days    = (options[:backfill_days]  || BACKFILL_DAYS_LIMIT).to_i

  Rails.logger.info '[DailyCallRecordTranscriptionWorker] Starting catch-up transcription run'

  new_calls_count = process_new_calls
  too_short_count = reconcile_too_short
  no_audio_count  = reconcile_no_audio
  fileless_count  = reconcile_fileless_upload
  backfill_count  = backfill_enabled ? backfill_older_calls(backfill_limit, backfill_days) : 0

  Rails.logger.info '[DailyCallRecordTranscriptionWorker] Completed: ' \
                    "#{new_calls_count} stuck/missed calls queued, #{too_short_count} marked too_short, " \
                    "#{no_audio_count} marked no_audio, #{fileless_count} fileless-upload marked no_audio, " \
                    "#{backfill_count} backfilled"
end