Module: DailyFocus::GroundingCheck

Defined in:: app/services/daily_focus/grounding_check.rb

Overview

Resolves every record a briefing points at, and reports the ones that do not
exist.

On 2026-08-10 a briefing told a rep to phone three customers, citing three
case numbers in a format we do not use and three CRM ids six times past the
end of the table (Basecamp 10137434670). The prompt already forbade it; a
prohibition inside a 17KB prompt is not a guardrail.

This is deliberately NOT a second model pass. A model checking a model
against the same data can agree with a fabrication, costs another
generation, and gives a different verdict on a rerun. Every fabrication seen
so far is a reference — a case number, a record id, a URL — and a
reference either resolves against the database or it does not. That check is
free, instant and the same every time.

It cannot judge narrative ("the customer said the mat reads 3 ohms"). The
defence there is upstream: sections built from data we hold (see
CoverageSection, FollowUpSection) leave the model nothing to invent.

It also answers the opposite question — did the briefing reproduce what we
precomputed? A section that is silently ABSENT resolves no bad references
and so passes every check above, which is exactly how conversation 4681
shipped Section 1 as prose with no coverage table at all.

Defined Under Namespace

Classes: Finding

Constant Summary collapse

LINKABLE = Resources whose records we can verify. A link to anything else passes silently — see #missing_records for why that is the default.

{
  'support_cases' => ->(ids) { SupportCase.where(id: ids).pluck(:id) },
  'article_technicals' => ->(ids) { Article.where(id: ids).pluck(:id) },
  'articles' => ->(ids) { Article.where(id: ids).pluck(:id) },
  'room_configurations' => ->(ids) { RoomConfiguration.where(id: ids).pluck(:id) },
  # The sales briefing's vocabulary, derived from what the prompts and
  # tools actually emit rather than from memory.
  'customers' => ->(ids) { Party.where(id: ids).pluck(:id) },
  'opportunities' => ->(ids) { Opportunity.where(id: ids).pluck(:id) },
  'quotes' => ->(ids) { Quote.where(id: ids).pluck(:id) },
  'orders' => ->(ids) { Order.where(id: ids).pluck(:id) },
  'parties' => ->(ids) { Party.where(id: ids).pluck(:id) },
  'employees' => ->(ids) { Party.where(id: ids).pluck(:id) }
}.freeze

LINK_PATTERN = Any CRM link in the briefing body, captured whole. Matching the path in one piece rather than as resource-then-id is what makes an unservable shape reportable: a narrower pattern simply fails to match /support-cases/123 or /support_cases/1/invalid, and a link nobody checks is the hole this class exists to close.

%r{#{Regexp.escape(CRM_URL)}/([^\s)\]]+)}

RECORD_PATH = A record link: one resource segment, one id, nothing after it. A record link: one resource segment, one id, nothing after it — or a nested child, which the briefing itself emits for an article revision (/articles/2038/revisions/5895). Flagging a URL we generate is the clearest possible sign the check is wrong, and it did exactly that.

%r{\A([a-z_]+)/(\d+)(?:/[a-z_]+/\d+)?\z}

CASE_LINK_PATTERN = A markdown link whose anchor is a case number and whose target is a case. Both halves can be real and still disagree, which is how a briefing sends a rep to the wrong case while looking correct.

/\[([A-Z]{2,4}\d{4,})\]\(#{Regexp.escape(SUPPORT_CASE_CRM_BASE_URL)}(\d+)\)/

COVERAGE_ROW = A row of the precomputed coverage table, keyed on the time cell — the one part of a row that cannot be reworded.

/^\|\s*(\d{1,2}:\d{2}\s[AP]M)\s+[–-]\s+\d{1,2}:\d{2}\s[AP]M\s*\|/

STATS_ROW = A row of the precomputed statistics table (StatsSection), captured whole: unlike a coverage row, the model writes no cell of it, so the entire row must survive — a kept label over a changed number is precisely the fabrication Section 4 exists to prevent.

/^\|\s*(?:Calls presented|Calls taken|Calls missed|Calls abandoned|Avg wait|Avg talk)\s*\|.*\|$/

CASE_NUMBER_PATTERN = Case numbers as the CRM writes them. Anchored on the four prefixes that actually exist (33,351 TTK, 367 ETK, 255 STK, 153 ATK) rather than any two-to-four letters: the loose form also matched CN/ON/SO/SQ, so every sales briefing citing a REAL customer or order would have been reported as "no case carries this number" — and findings nobody can trust are findings nobody reads.

/\b((?:TTK|ETK|STK|ATK)\d{4,})\b/

REFERENCE_PATTERN = The CRM's other reference prefixes, each resolved against its own table. CN is the party id, SO the order id, SQ the quote id; ON is an opportunity, matched on reference_number first and id second, the same order Crm::OpportunityLinkPath resolves them in. One digit up, not four: party ids start at 1 and opportunity ids at 46, so a four-digit floor silently exempted every small record from the check. The # or the absence of a space is what keeps prose out — "ON 5 August" does not match, "ON5" and "ON #5" do.

/\b(CN|ON|SO|SQ)(?:\s?#\s?|#?)(\d+)\b/

REFERENCE_RESOLVERS = CN is the only one that is a row id — a customer number IS the party id. SO, SQ and ON are reference_numbers ("SO729425" on order 1396080), which do not equal the id and are what the CRM shows a rep. Resolving those against ids reported every genuine order and quote as fabricated. Returns: (Hash{String => Proc}) — prefix => resolver returning the ids that exist

{
  'CN' => ->(ids) { Party.where(id: ids).pluck(:id) },
  'SO' => ->(ids) { GroundingCheck.send(:existing_reference_numbers, Order, 'SO', ids) },
  'SQ' => ->(ids) { GroundingCheck.send(:existing_reference_numbers, Quote, 'SQ', ids) },
  'ON' => ->(ids) { GroundingCheck.send(:existing_reference_numbers, Opportunity, 'ON', ids) }
}.freeze

REFERENCE_LABELS = What each prefix is called when a briefing cites one that does not exist.

{ 'CN' => 'customer', 'ON' => 'opportunity', 'SO' => 'order', 'SQ' => 'quote' }.freeze

FATAL_KINDS = The finding kinds that ARE fabrication: a reference resolving to no record, or to the wrong one. These block a briefing (the worker retries once, then fails it) — measured across 2026-07-01..08-13, invented references ran 0–4.5% of citations on gemini-3.5, 10–60% on gemini-3.6 and 5.6% on GLM 5.2, and every one shipped to the review queue as ready. The remaining kinds are degradation (a dropped table row, a closed case cited as live work): visible on the review row, not fatal, because a manager rejecting one stale row beats binning a briefing.

%i[missing_record missing_case_number missing_reference case_link_mismatch].freeze

Class Method Summary collapse

.call(content, prompt: nil, rep_ids: []) ⇒ Array<Finding>
Empty when every reference resolves, every precomputed row survived, and every named case is live work.

Class Method Details

.call(content, prompt: nil, rep_ids: []) ⇒ `Array<Finding>`

Returns empty when every reference resolves, every
precomputed row survived, and every named case is live work.

Parameters:

content (String) —
the finished briefing markdown
prompt (String, nil) (defaults to: nil) —
the turn's user message, carrying the
precomputed sections the briefing was supposed to reproduce
rep_ids (Array<Integer>) (defaults to: []) —
the recipient plus anyone they cover, so a
named case can be judged against their live work

Returns:

(Array<Finding>) —
empty when every reference resolves, every
precomputed row survived, and every named case is live work

# File 'app/services/daily_focus/grounding_check.rb', line 125

def self.call(content, prompt: nil, rep_ids: [])
  return [] if content.blank?

  unresolvable_links(content) + unresolvable_case_numbers(content) +
    unresolvable_references(content) + mismatched_case_links(content) +
    dropped_coverage_rows(content, prompt) + altered_stats_rows(content, prompt) +
    misattributed_cases(content, rep_ids)
end