Class: Assistant::TechnicalSupport::CaseReplayAnalyzer

Inherits:
Object
  • Object
show all
Defined in:
app/services/assistant/technical_support/case_replay_analyzer.rb

Overview

Uses a bounded structured-output pass to split a resolved case into
observable replay facts and a hidden verified outcome, then a separate
pass to compare Sunny's answer with that outcome. The orchestration layer
independently verifies fragment separation and citation provenance.

Defined Under Namespace

Classes: InvalidResponse, Result

Constant Summary collapse

EPISODE_SCHEMA = 
{
  type: 'object',
  properties: {
    eligible: { type: 'boolean' },
    ineligible_reason: { type: %w[string null] },
    product_family: { type: %w[string null] },
    issue_summary: { type: %w[string null] },
    opening_fragment_ids: { type: 'array', items: { type: 'string' } },
    follow_up_fragment_ids: { type: 'array', items: { type: 'string' } },
    outcome_fragment_ids: { type: 'array', items: { type: 'string' } },
    verified_outcome: { type: %w[string null] },
    required_facts: { type: 'array', items: { type: 'string' } },
    safety_requirements: { type: 'array', items: { type: 'string' } },
    article_search_query: { type: %w[string null] }
  },
  required: %w[
    eligible ineligible_reason product_family issue_summary
    opening_fragment_ids follow_up_fragment_ids outcome_fragment_ids
    verified_outcome required_facts safety_requirements article_search_query
  ],
  additionalProperties: false
}.freeze
FRAGMENT_ID_FIELDS = 
%i[
  opening_fragment_ids follow_up_fragment_ids outcome_fragment_ids
].freeze
OUTCOME_FRAGMENT_ID_FIELD = 
:outcome_fragment_ids
MAX_SCHEMA_FRAGMENT_IDS = 
500
EVALUATION_SCHEMA = 
{
  type: 'object',
  properties: {
    benchmark_grounded: { type: 'boolean' },
    benchmark_evidence_ids: { type: 'array', items: { type: 'string' } },
    same_outcome: { type: 'boolean' },
    grounded: { type: 'boolean' },
    safe: { type: 'boolean' },
    missing_facts: { type: 'array', items: { type: 'string' } },
    unsupported_claims: { type: 'array', items: { type: 'string' } },
    article_gap: { type: %w[string null] },
    notes: { type: %w[string null] }
  },
  required: %w[
    benchmark_grounded benchmark_evidence_ids same_outcome grounded safe
    missing_facts unsupported_claims article_gap notes
  ],
  additionalProperties: false
}.freeze

Instance Method Summary collapse

Constructor Details

#initialize(chat_factory: nil) ⇒ CaseReplayAnalyzer

Builds an analyzer, optionally with an injected model-chat factory.

Parameters:

  • chat_factory (Proc, nil) (defaults to: nil) —

    factory receiving the selected model ID



83
84
85
 
# File 'app/services/assistant/technical_support/case_replay_analyzer.rb', line 83

def initialize(chat_factory: nil)
  @chat_factory = chat_factory || method(:build_chat)
end

Instance Method Details

#evaluate(episode:, replay_facts:, outcome_facts:, answer:, tool_trace:) ⇒ Result

Evaluates Sunny's answer against the hidden verified outcome.

Parameters:

  • episode (CaseReplay::Episode)
  • replay_facts (Array<Hash>) —

    typed facts Sunny was shown

  • outcome_facts (Array<Hash>) —

    hidden raw outcome evidence

  • answer (String)
  • tool_trace (Array<CaseReplay::ToolCall>)

Returns:

  • (Result) —

    successful schema-constrained replay assessment



117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
 
# File 'app/services/assistant/technical_support/case_replay_analyzer.rb', line 117

def evaluate(episode:, replay_facts:, outcome_facts:, answer:, tool_trace:)
  Result.new(
    status: :success,
    data: ask(
      schema_name: 'TechnicalSupportCaseReplayEvaluation',
      schema: evaluation_schema(outcome_facts),
      prompt: CaseReplayEvaluationPrompt.render(
        episode:,
        replay_facts:,
        outcome_facts:,
        answer:,
        tool_trace:
      ),
      feature: 'technical_support_case_replay_evaluation'
    )
  )
end

#extract(events:, coverage:) ⇒ Result

Extracts a replayable episode from the complete de-identified stream.

Parameters:

Returns:

  • (Result) —

    eligible episode or expected ineligible outcome



92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
 
# File 'app/services/assistant/technical_support/case_replay_analyzer.rb', line 92

def extract(events:, coverage:)
  data = ask(
    schema_name: 'TechnicalSupportCaseEpisode',
    schema: episode_schema(events),
    prompt: CaseEpisodePrompt.render(events:, coverage:),
    feature: 'technical_support_case_episode'
  )

  return Result.new(status: :success, data:) if data['eligible'] == true

  Result.new(
    status: :ineligible,
    data:,
    reason: data['ineligible_reason'].to_s.strip.presence
  )
end