Skip to content
Heatwave Docs

Daily Focus Re-run Runbook

Audience: engineer on call for the CRM / Sunny stack. When: the morning Daily Focus briefings failed, generated garbage, or are stuck in “Processing”, and you need to regenerate them for today. Why it needs a runbook: a naive re-run of the orchestrator silently does nothing in three separate ways. Each one looks like “the job ran and no briefing appeared.”

Design and feature background (roles, coverage policy, support-rep variant) is in doc/tasks/202607161719_DAILY_FOCUS_SUPPORT_REPS.md. That is a design doc, not a procedure — this is the procedure.

What normally runs

Section titled “What normally runs”
Piece Schedule (America/Chicago) Does
DailyFocusAnalysisWorker 0 8 * * 1-5 Fans out one DailyFocusAnalysisRepWorker per RunPolicy target (sales + support), staggered index * 30s, then DailyFocusManagerReportWorker after 20 min
DailyFocusAnalysisRepWorker on demand Creates/reclaims the rep’s briefing and generates it through DailyFocus::ChatRunner
DailyFocusOrphanReaperWorker */10 5-18 * * 1-5 Re-enqueues briefings stranded in processing past ORPHAN_AFTER

A briefing is an AssistantConversation with metadata conversation_type='daily_briefing', daily_focus_date, daily_focus_status (processing / ready / failed), daily_focus_target_employee_id, daily_focus_covered_employee_ids, daily_focus_audience. Manager approval (DailyFocusController#approve) strips daily_focus_target_employee_id and daily_focus_status and sets user_id to the rep.

Before you touch anything: triage

Section titled “Before you touch anything: triage”

Run in a production Rails runner (see “Running commands in production” below):

convs = AssistantConversation.for_daily_focus_date(Date.current)
convs.map { |c|
  { id: c.id,
    rep: c.daily_focus_target_employee_id,
    user: c.user_id,
    status: c.daily_focus_status,      # nil => APPROVED
    audience: c.daily_focus_audience,
    updated: c.updated_at,
    live: DailyFocus.briefing_live?(c) }
}

Read the result against this table before choosing a step.

daily_focus_status target_employee_id State Will a plain re-run regenerate it?
ready set Pending manager review No — briefing_complete?
nil nil Approved (already emailed) No — briefing_complete?
processing set, updated_at < 15 min ago Live or just-fired No — briefing_live?
processing set, updated_at > 15 min ago Orphaned Yes (reaper does it too)
failed set Failed Yes
— — No row at all Yes

“The briefing said something wrong” — read the prompt, not the output

Section titled ““The briefing said something wrong” — read the prompt, not the output”

Most reports of a bad briefing are a rendering fault, not a data fault, and the two are told apart in one query. A conversation holds the prompt as its first user message and the briefing as its last assistant message, so the replica answers “did we hand it the wrong facts, or did it mangle the right ones?” without touching production:

Terminal window
FOCUS_DATE=2026-08-10        # the day the rep is complaining about


# 1. find the conversation ids for that day
psql "$POSTGRES_REPLICA_URI" -v ON_ERROR_STOP=1 -v focus_date="'$FOCUS_DATE'" <<'SQL'
SET default_transaction_read_only = on;
SELECT ac.id, p.full_name, ac.metadata->>'daily_focus_status' AS status
FROM assistant_conversations ac
LEFT JOIN parties p ON p.id = (ac.metadata->>'daily_focus_target_employee_id')::bigint
WHERE ac.metadata->>'daily_focus_date' = :focus_date;
SQL


CONV=4647                    # one id from the list above


# 2. what we handed it — Section 1 arrives precomputed, so this is ground truth
psql "$POSTGRES_REPLICA_URI" -v ON_ERROR_STOP=1 -v conv="$CONV" <<'SQL'
SET default_transaction_read_only = on;
SELECT substring(content FROM position('SECTION 1 — ROLLING QUEUE COVERAGE' IN content) FOR 2500)
FROM assistant_messages WHERE assistant_conversation_id = :conv AND role = 'user' ORDER BY id ASC LIMIT 1;
SQL


# 3. what it produced
psql "$POSTGRES_REPLICA_URI" -v ON_ERROR_STOP=1 -v conv="$CONV" <<'SQL'
SET default_transaction_read_only = on;
SELECT content FROM assistant_messages
WHERE assistant_conversation_id = :conv AND role = 'assistant' ORDER BY id DESC LIMIT 1;
SQL

Step 2’s marker tracks DailyFocus::CoverageSection.prompt_section — if it returns nothing, read that method for the current heading rather than assuming the section is missing.

Compare two reps’ briefings for the same day before concluding anything: the coverage table is a team view, so any disagreement between them is a rendering bug by definition. That comparison is what found the 2026-08-10 batch — the prompt had Chris’s lunch, both briefings lost it differently (Basecamp 10137434670).

Procedure

Section titled “Procedure”

1. Delete anything that counts as “complete”

Section titled “1. Delete anything that counts as “complete””

DailyFocusAnalysisRepWorker#briefing_complete? is daily_focus_status == 'ready' || daily_focus_target_employee_id.nil?. An approved briefing has a nil target, so it counts as complete and is skipped — even when its content is a failure message that a manager approved anyway. There is no force flag. You must delete first.

This mirrors DailyFocusController#reject:

ids = [...]   # from triage; be explicit, never a blanket destroy_all
AssistantConversation.where(id: ids).find_each do |c|
  c.release_processing_lock! if c.processing_by_id.present?
  c.destroy!
end

destroy! cascades the conversation’s messages and shares. Deleting also clears gotcha 3 below, which is why it’s the deterministic option.

2. Re-fire

Section titled “2. Re-fire”

Whole day, all reps — same path as the 8 AM cron:

DailyFocusAnalysisWorker.perform_async

It applies the 30-second stagger itself.

Specific reps — stagger them yourself, 60s apart:

[rep_id_1, rep_id_2, rep_id_3].each_with_index do |rep_id, i|
  DailyFocusAnalysisRepWorker.perform_in(i * 60, rep_id, nil, [])
end

Pass nil for conversation_id so the worker self-creates under its per-rep advisory lock (daily_focus:<employee_id>:<date>) and reclaims orphans. The third argument is covered_employee_ids — carry over the ids from the deleted briefing if the rep was covering for someone.

⚠️ Passing a real conversation_id bypasses ensure_conversation_for entirely — no complete check, no live check, no advisory lock. That’s the CRM “Generate” button’s path (DailyFocusController#generate), which is fine because it just created the conversation. Don’t reuse it for a re-run.

3. Watch

Section titled “3. Watch”
AssistantConversation.for_daily_focus_date(Date.current)
                     .map { |c| [c.id, c.daily_focus_target_employee_id, c.daily_focus_status] }

A healthy generation takes up to ~7 minutes and keeps bumping updated_at as PlanOrchestrator rewrites the execution_plan metadata. Fresh briefings land ready — pending review, not sent. A manager re-approves each one from the CRM Daily Focus review page (/daily_focus/review), which re-sends InternalReportsMailer.daily_focus_approved. Reps who were already emailed from the first (bad) approval will get a second email.

4. If some reps stall

Section titled “4. If some reps stall”

Wait for DailyFocusOrphanReaperWorker (every 10 min, 05:00–18:00 weekdays) — it re-enqueues anything in processing with updated_at older than 15 minutes, preserving covered reps. If you don’t want to wait, go back to step 1 for those conversations and re-fire them individually with the 60-second stagger.

Why it silently did nothing

Section titled “Why it silently did nothing”
Symptom Cause Fix
Job ran, logs show “Enqueuing daily focus for N targets”, but a rep’s briefing is unchanged briefing_complete? — the briefing is ready, or approved (target id stripped by #approve, so nil ⇒ complete) Delete it first (step 1). No force flag exists.
Re-fired a stuck briefing, worker logged nothing and exited DailyFocus.briefing_live? — processing? or updated_at within DailyFocus::ORPHAN_AFTER (15 min). The worker hits next nil and skips without reclaiming. Wait >15 min, or destroy the conversation explicitly and re-fire
Fired everyone at once; several briefings frozen in processing with processing_by_id nil and a stale execution_plan Sidekiq box OOM’d under ~8 concurrent LLM generations and hard-killed workers. Sidekiq::Shutdown is an Interrupt, not a StandardError, so the worker’s rescue never marked them failed. Stagger re-fires ~60s apart. Recover the stranded ones via the reaper or step 1.
Everything failed at once with “Step failed” Upstream model/registry problem, not the briefing pipeline (DailyFocus::CHAT_MODEL_KEY is gemini-flash; a missing entry raises ModelNotFoundError that surfaces as a failed step) Fix the model config first, then run this procedure

Running commands in production

Section titled “Running commands in production”
Terminal window
kamal app exec --reuse --roles=web "bin/rails runner \"eval(Base64.decode64('<b64>'))\""
  • Pin --roles=web. Without it kamal app exec runs on every role, so a perform_async enqueues once per role.
  • Base64-encode the runner script to dodge nested shell quoting — multi-line Ruby through kamal app exec mangles otherwise.
  • bin/deploy and rails console are hard-blocked for agents; ask before running anything here.

Reference

Section titled “Reference”
Thing Where
ORPHAN_AFTER, briefing_live?, CHAT_MODEL_KEY, rep_scope app/services/daily_focus.rb
briefing_complete?, ensure_conversation_for, reclaim app/workers/daily_focus_analysis_rep_worker.rb
Fan-out + 30s stagger app/workers/daily_focus_analysis_worker.rb
Orphan sweep app/workers/daily_focus_orphan_reaper_worker.rb
approve / reject / generate app/controllers/daily_focus_controller.rb
Metadata scopes app/models/assistant_conversation.rb
Cron entries config/sidekiq_production_schedule.yml (daily_focus_analysis_worker, daily_focus_orphan_reaper)