Class: Assistant::ChatService

Inherits:
Object
  • Object
show all
Includes:
PromptComposer
Defined in:
app/services/assistant/chat_service.rb

Overview

Service for AI-powered assistant chat using RubyLLM's acts_as_chat.
Uses tool-based architecture: the LLM calls registered tools (DB, content search, etc.)
rather than generating raw SQL. Conversation history is managed by RubyLLM automatically.

Defined Under Namespace

Classes: Result

Constant Summary collapse

THINKING_BUDGET_LOW =

Extended Thinking configuration — gives reasoning models a scratchpad
for multi-step problems (SQL construction, analytical reasoning).
Legacy Anthropic models use token budgets; current Gemini uses effort
levels and does not receive these budget values.

4_000
THINKING_BUDGET_MEDIUM =

Simple tool queries

8_000
THINKING_BUDGET_HIGH =

Analytical queries with JOINs/aggregation

16_000
THINKING_BUDGET_MAX =

Complex multi-step reasoning (Opus only)

64_000
EFFORT_RANK =

Relative ordering of thinking-effort tiers, lowest → highest. Lets the
router pick the higher of a query-driven floor and a model's configured
default, so a :max-default model is never capped at :high.

{ low: 0, medium: 1, high: 2, max: 3 }.freeze
THINKING_QUERY_PATTERNS =

Patterns that indicate the query would benefit from extended thinking

/\b(compare|analyze|trend|correlat|calculate|forecast|predict|why|root.?cause|deep.?dive|break.?down|step.?by.?step|optimize|investigate|audit|reconcil|year.?over.?year|month.?over.?month)\b/i
LLM_NETWORK_RETRY_EXCEPTIONS =

Transient provider / TLS failures (AppSignal #4527: Faraday::SSLError SSL_read EOF).

[
  Faraday::SSLError,
  Faraday::ConnectionFailed,
  Faraday::TimeoutError,
  OpenSSL::SSL::SSLError,
  RubyLLM::ServiceUnavailableError, # HTTP 502/503/504 — upstream gateway transient
  RubyLLM::OverloadedError          # HTTP 529 — Anthropic "service overloaded" transient
].freeze
MODELS =

Available models with their configurations.
Model IDs come from AiModelConstants — the single source of truth.
supports_thinking: whether the model supports RubyLLM's with_thinking (extended reasoning)
thinking_effort_default: the default effort level when thinking is activated (:low, :medium, :high, :max)

{
  'claude-haiku'  => { id: AiModelConstants.id(:anthropic_haiku),  provider: :anthropic, label: 'Claude Haiku 4.5 (Fast)', cost: :low, supports_thinking: false },
  'claude-sonnet' => { id: AiModelConstants.id(:anthropic_sonnet), provider: :anthropic, label: 'Claude Sonnet 5 (Balanced)', cost: :medium, supports_thinking: true, thinking_effort_default: :medium },
  'claude-opus'   => { id: AiModelConstants.id(:anthropic_opus),   provider: :anthropic, label: 'Claude Opus 5 (Best — highest cost)', cost: :high, supports_thinking: true, thinking_effort_default: :high },
  # Same Opus model, opened up to the 1M-token context window via the
  # Anthropic context-1m beta header (see configure_conversation). Top rung
  # of the complexity-escalation ladder for marathon / huge-context sessions.
  'claude-opus-1m' => { id: AiModelConstants.id(:anthropic_opus),  provider: :anthropic, label: 'Claude Opus 5 (1M context)', cost: :high, supports_thinking: true, thinking_effort_default: :high, context_1m: true },
  'gpt-5'         => { id: AiModelConstants.id(:openai_gpt5),      provider: :openai,    label: 'GPT-5 (OpenAI)',               cost: :medium, supports_thinking: false },
  'gpt-5.5'       => { id: AiModelConstants.id(:openai_gpt55),     provider: :openai,    label: 'GPT-5.5 (OpenAI Latest)',      cost: :medium, supports_thinking: false },
  'gpt-5-mini'    => { id: AiModelConstants.id(:openai_gpt5_mini), provider: :openai,    label: 'GPT-5 Mini (Fast)',            cost: :low,    supports_thinking: false },
  'gemini-flash'  => { id: AiModelConstants.id(:gemini_flash),     provider: :gemini,    label: 'Gemini 3.7 Flash (Recommended)', cost: :low, supports_thinking: true, thinking_effort_default: :low },
  'gemini-pro'    => { id: AiModelConstants.id(:gemini_pro),       provider: :gemini,    label: 'Gemini 3.7 Flash · High Reasoning (Google)', cost: :medium, supports_thinking: true, thinking_effort_default: :high },
  # Kimi K3 via the direct Moonshot API — open-weight opt-in preview.
  # Explicit picker choice ONLY: not reachable from auto_select_model,
  # absent from the ComplexityEscalator ladder, and opt_in: true keeps it
  # out of model affinity — switching the dropdown back to 'auto' leaves
  # Kimi immediately instead of "keeping it for context continuity".
  # Sonnet-level pricing ($3/$15 per Mtok) → :medium tier; automatic
  # server-side context caching bills stable prefixes at the cache-read
  # rate. K3 fixes sampling params server-side, so
  # AiModelConstants.rejects_sampling_params? skips with_temperature for
  # it. Requires kimi_ai.api_key in credentials; a failed call surfaces
  # in-chat — no silent fallback, so spend can't leak onto another model
  # unnoticed.
  'kimi-k3'       => { id: AiModelConstants.id(:kimi),             provider: :moonshotai, label: 'Kimi K3 (Moonshot · opt-in preview)', cost: :medium, supports_thinking: false, opt_in: true },
  # Grok 4.5 via native xAI API — opt-in preview. Same affinity / auto-
  # select isolation as kimi-k3. Strong agentic/tool-calling pitch at
  # $2/$6 per Mtok (under Sonnet list, far under Opus) with 500k context
  # and $0.30 cache-read on stable prefixes. Reasoning is available on
  # the API; RubyLLM's with_thinking path for xAI is not wired the same
  # as Anthropic adaptive thinking yet, so supports_thinking stays false
  # until we prove the effort param. Requires xai_api.api_key.
  'grok-4.6'      => { id: AiModelConstants.id(:grok),             provider: :xai,        label: 'Grok 4.6 (xAI · opt-in preview)', cost: :medium, supports_thinking: false, opt_in: true },
  # Open-weight test fleet via OpenRouter — same opt-in isolation as
  # kimi-k3/grok-4.6 (explicit picker choice only; no auto-select, no
  # escalation ladder, no model affinity). One gateway for all three so
  # the briefing evaluation is apples-to-apples: shared privacy prefs +
  # session pinning (apply_openrouter_provider_options!), Auto Exacto
  # host ordering on tool calls, exact usage.cost accounting. Pricing
  # and rationale: AiModelConstants + doc/development/OPENROUTER.md.
  # supports_thinking stays false until the OpenRouter reasoning param
  # is proven per model (same stance as grok-4.6).
  # fallback_models: OpenRouter's server-side `models` array — on any
  # error (model outage, empty eligible pool, rate limit) the request
  # advances to the next model, billed at whichever served, with the
  # same provider prefs applied. Host-level failover WITHIN a model
  # needs nothing here — OpenRouter load balances GLM's ~30 endpoints
  # on its own. The response's `model` field reports what actually
  # answered and RubyLLM parses it into the message, so ai_usage_logs
  # prices a fallback day as whoever ran.
  #
  # Kimi K2.6, NOT the A/B runner-up Qwen3.7 Plus: adopting zdr: true
  # (below) narrows every request to attested zero-retention endpoints,
  # and Qwen publishes exactly one endpoint, which isn't one — the
  # array would have been unroutable on the day it was needed. Measured
  # 2026-08-12 against /api/v1/endpoints/zdr: GLM 21 eligible hosts
  # (of 32), Kimi 15 (of 21), Qwen 0 (of 1) — counting only hosts that
  # also declare tool_choice, since we send require_parameters.
  'glm-5.2'       => { id: AiModelConstants.id(:openrouter_glm),  provider: :openrouter, label: 'GLM 5.2 (Z.AI · opt-in preview)', cost: :low, supports_thinking: false, opt_in: true,
                       fallback_models: [AiModelConstants.id(:openrouter_kimi)] },
  'qwen3.7-plus'  => { id: AiModelConstants.id(:openrouter_qwen), provider: :openrouter, label: 'Qwen3.7 Plus (Alibaba · opt-in preview)', cost: :low, supports_thinking: false, opt_in: true },
  'kimi-k2.6'     => { id: AiModelConstants.id(:openrouter_kimi), provider: :openrouter, label: 'Kimi K2.6 (Moonshot via OpenRouter · opt-in preview)', cost: :low, supports_thinking: false, opt_in: true }
}.freeze
DEFAULT_MODEL =

Default model.

'gemini-flash'
CONTEXT_1M_BETA =

Anthropic beta token that unlocks Opus's 1M-token context window, applied
only to the 'claude-opus-1m' model (context_1m: true) via with_headers.
Validated live against api.anthropic.com on 2026-06-03 — accepted (HTTP 200),
as was adaptive thinking at effort=max on the Opus tier (now claude-opus-5).

'context-1m-2025-08-07'
MAX_PLAN_COST_USD =
Note:

plan_cost underestimates because run_plan_step_executor returns only the FINAL
API round's tokens (not the cumulative total across tool-call rounds within a step).
Real per-step cost is typically 5-10× higher than reported. The primary cost guard is
the ToolLoopGuard's per-step call limit, not this cap.

Hard cap on estimated plan execution cost (USD) across isolated step + assembly LLM calls.

2.00
MAX_PLAN_STEP_DURATION =

Wall-clock timeout per plan step — driven from ToolLoopGuard so both
the outer Timeout and the inner guard share a single source of truth.

Assistant::ToolLoopGuard::MAX_STEP_DURATION.seconds
STEP_RESULT_MAX_CHARS =

How much of a plan step's result travels to the next step and the final
assembly. Named for what it does: nothing summarizes the overflow, it is
simply cut (see PlanOrchestrator#clamp_step_result_for_chain). The old
name — STEP_RESULT_SUMMARIZE_THRESHOLD — described a cheap-model
summarization pass that does not exist, which is how 2,000 chars looked
survivable for years.

2,000 discarded most of every data-gathering step. Measured against the
queries the sales briefing actually runs for one rep (conv 4843,
2026-08-14): 23 quote follow-ups render 8,609 chars and 10 lapsed trade
accounts 2,645 — 11,254 for the single step that runs both, so ~83% was
cut and the briefing told the rep its own data was "truncated in the tool
output". The support briefing hit the same wall from the other side: nine
consolidated queries were cut mid-table right before the call statistics,
and the assembly filled the hole by inventing numbers (conv 4821).

12,000 clears that measured worst case with headroom. The ceiling on
accumulation is per-step, so a 10-step plan tops out around 120k chars
(~30k tokens) of step results — comfortable for the 128k+ context models
that run plans, and MAX_PLAN_COST_USD still bounds spend independently.

12_000
MID_TURN_COMPACT_THRESHOLD =

Mid-turn compaction thresholds (see install_mid_turn_compaction!)

2_000
MID_TURN_KEEP_CHARS =

Mid turn keep chars.

600
MID_TURN_COMPACT_AFTER_MESSAGES =

Messages (assistant + tool, not rounds) before the mid-turn pass arms.

3
MID_TURN_MIN_MESSAGES =

Below this transcript length there is nothing worth compacting.

6
MID_TURN_SKIP_PREFIXES =

Content already shrunk by another pass. '[Compacted' / '[Truncated' are
the pre-2026-08 raw-cut formats — kept so a transcript replayed across
the deploy isn't compacted twice; new output is JSON carrying _truncated
and is skipped by MID_TURN_COMPACT_THRESHOLD instead.

['[Compacted', '[Truncated', '[Already retrieved'].freeze
COMPLEX_QUERY_PATTERNS =

Keywords indicating complex analytical or reasoning queries (need better models)

/\b(why|trend|pattern|anomaly|recommend|insight|correlation|predict|forecast|explain|root.?cause|deep.?dive|strategic|analyze|summarize|evaluate|pros?.and.cons|trade.?off)\b/i
COMPARISON_QUERY_PATTERNS =

Keywords indicating multi-step comparison or research queries (need balanced models)

/\b(compare|vs|versus|between|difference|change|growth|decline|year.?over.?year|month.?over.?month|yoy|mom|research|investigate|audit)\b/i
SIMPLE_QUERY_PATTERNS =

Keywords indicating simple lookup or factual queries (fast models are fine)

/\b(show|list|get|total|count|how many|what is|what are|sum|average|find|look up|search|where is|who is|when did)\b/i
COMPOSE_QUERY_PATTERNS =

Phrases that indicate the user is drafting/composing a short message (email,
follow-up, outreach, internal summary). These are quick content-generation
tasks where Flash is fast and good enough — Pro's extended thinking is
wasted budget here, and on long prompts (e.g. pasted email threads) we'd
otherwise route them to Pro and time out.

/\b(reply|respond|send|email|follow.?up|outreach|reach out|thank.?you note|summary email)\b/i
WRITING_QUERY_PATTERNS =

Phrases that indicate long-form editorial work (blog posts, articles, FAQs,
rewrites). Flash produces noticeably weaker prose here — see /assistant/1639,
where a Buffalo bathroom blog post written under Flash drew "wrote very poorly"
feedback from the editor. Content-authoring tasks now route to Claude Sonnet:
every Gemini tier proved slow and unreliable on long HTML body edits — the
old gemini-3.1-pro preview intermittently 400'd (#3808) and ground out the
full 600s plan-step timeout on complex edits (#4714, conv 3098), which is why
the Gemini Pro snapshots were dropped from the registry entirely.
Opus is intentionally excluded as too expensive for routine editorial work.

Regexp.new(
  '\b(rewrite|polish|copyedit|copy.?edit|long.?form|article|blog post|blog ?article|blog ?entry|' \
  'essay|narrative|edit blog|write the blog|draft the blog|update the blog|update the article|' \
  'expand this section|tighten this|story|landing page copy|product description|press release|' \
  'case study|whitepaper|white ?paper|content brief|seo copy|meta description|page copy|' \
  'h(?:ero|eading) copy|body copy|email template|email campaign|email blast|email copy|' \
  'email design|newsletter|technical.?articles?|troubleshooting (?:guides?|sources?)|' \
  'knowledge consolidation)\b',
  Regexp::IGNORECASE
)
CONTENT_AUTHORING_SERVICES =

Tool services whose presence marks a content-authoring turn. When the
classifier routes a turn to these, it gets Claude regardless of the query
wording (covers follow-ups like "now add a CTA" that lack writing keywords).

%w[
  blog_management
  email_management
  technical_article_management
].freeze
EXTENDED_TURN_SERVICES =

Services whose turns legitimately run long and emit large output, so they
get MAX_TURN_DURATION_AUTHORING (600s) and the 32k output budget rather
than the 360s telemetry cap.

A superset of CONTENT_AUTHORING_SERVICES on purpose. pdf_tools belongs on
the budget list — pdf_generate assembles a whole document across many tool
calls and was being halted mid-build at 430s against the 360s cap
(AppSignal #6189) — but must NOT join CONTENT_AUTHORING_SERVICES, which
additionally forces the turn onto Claude regardless of query wording.
Widening that constant would have quietly changed PDF model routing.

(CONTENT_AUTHORING_SERVICES + %w[pdf_tools]).freeze
WRITING_MODEL_DEFAULT =

Model we auto-route content-authoring work to. The one place we deliberately
auto-pick Anthropic — Claude is materially more reliable + faster at HTML
body editing than any Gemini tier. Opus stays opt-in (cost)
for general editorial; blog editing is the exception — see BLOG_AUTHORING_MODEL.

'claude-sonnet'
WRITING_MODEL_CLAUDE =

Writing model claude.

'claude-sonnet'
BLOG_AUTHORING_MODEL =

Blog editing is the heaviest content-authoring workload: large HTML bodies,
many block-level tool calls, long multi-turn sessions. On Gemini — and even
Sonnet — these turns repeatedly tripped the body-less Gemini 400 (#3808) and
the 600s plan-step timeout (#4714), and large posts got shredded by mid-turn
compaction — leaving the model editing from truncated HTML and looping until
it timed out (convs 3105/3109, Julia). Route blog editing to Opus 5 on the
1M-token context window from the FIRST turn so the model has both the
capability and the context headroom to finish without choking, instead of
starting cheap and escalating only after it has already failed. Cost is the
deliberate tradeoff for blog work specifically — email/general editorial
stay on Sonnet. Defined as a constant so the tier is easy to retune.

'claude-opus-1m'
GATEWAY_FALLBACK_MODEL =

Where an OpenRouter turn goes when the gateway itself is unreachable —
a different provider on purpose, so a single vendor outage can't take the
briefing down. See apply_gateway_fallback! for why it isn't gemini-flash.

'claude-sonnet'
BLOG_AUTHORING_SERVICES =

Classifier tool services that mark a blog authoring turn (vs. email).

%w[blog_management].freeze
BLOG_AUTHORING_PATTERNS =

Query wording that signals blog editing even without a classifier tool hint
(e.g. tests, or a turn the classifier abstained on). Deliberately blog-ONLY:
generic "article"/"the article" wording is left to content_authoring_turn? →
Sonnet, so a plain editorial edit isn't forced onto the pricier Opus-1M tier.

Also matches a pasted WarmlyYours blog-post URL (…/posts/) and the
"for this/the blog" lead-in: the common way an editor kicks off a blog task
is to paste the post URL ("for this blog https://…/posts/…/preview"), which
carries no other blog keyword and otherwise fell through to Gemini and hit
the intermittent body-less 400 (#3808, conv 3150). %r{} so the /posts/ path
needs no escaping.

%r{\b(blog post|blog ?article|blog ?entry|edit (?:the )?blog|update (?:the )?blog|write (?:the )?blog|draft (?:the )?blog|rewrite (?:the )?blog(?: post| article| entry)?|for (?:this|the) blog)\b|/posts/[\w-]+}i
WRITING_ELIGIBLE_MODELS =

Writing eligible models.

[WRITING_MODEL_DEFAULT, WRITING_MODEL_CLAUDE, BLOG_AUTHORING_MODEL].uniq.freeze
MODEL_COST_TIER =

Cost tiers for model affinity decisions.
Switching models mid-conversation loses accumulated reasoning context,
so we only switch when escalating to a higher tier (never laterally).

MODELS.transform_values { |c| c[:cost] }.freeze

Constants included from PromptComposer

PromptComposer::AGENT_PROMPTS_DIR, PromptComposer::ANALYTICS_SERVICES, PromptComposer::DOMAIN_TOOL_REQUIREMENTS, PromptComposer::INSTRUCTIONS_TEMPLATE_PATH, PromptComposer::MESSAGE_DOMAIN_PATTERNS

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(conversation:, user_message:, model: 'auto', tool_services: [], permitted_services: [], user_context: {}, on_status: nil, cancel_check: nil, attachments: [], log_metadata: {}) ⇒ ChatService

Returns a new instance of ChatService.

Parameters:

  • conversation (AssistantConversation) —

    The conversation record (acts_as_chat)

  • user_message (String) —

    The user's query

  • model (String) (defaults to: 'auto') —

    LLM model key or 'auto'

  • tool_services (Array<String>) (defaults to: []) —

    Service keys for tool access

  • permitted_services (Array<String>) (defaults to: []) —

    All service keys the user's role allows (for tool suggestion prompt)

  • user_context (Hash) (defaults to: {}) —

    User identity for personalized queries

  • on_status (Proc) (defaults to: nil) —

    Callback for status events

  • cancel_check (Proc) (defaults to: nil) —

    Returns true when the caller wants to abort (e.g. user clicked Stop)

  • attachments (Array<Pathname>) (defaults to: []) —

    Optional file paths to attach to the message (PDFs, images, etc.)

  • log_metadata (Hash) (defaults to: {}) —

    extra key/value pairs passed to instrumentation and usage logs



507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
 
# File 'app/services/assistant/chat_service.rb', line 507

def initialize(conversation:, user_message:, model: 'auto', tool_services: [], permitted_services: [], user_context: {}, on_status: nil, cancel_check: nil, attachments: [], log_metadata: {})
  @conversation = conversation
  @user_message = user_message
  @log_metadata = log_metadata || {}
  @tool_services = Array(tool_services).compact_blank
  @permitted_services = Array(permitted_services).compact_blank
  @user_context = user_context || {}
  @on_status = on_status
  @cancel_check = cancel_check
  @attachments = Array(attachments).select do |p|
    if p.respond_to?(:exist?)
      p.exist?                      # Pathname — check local file
    elsif p.to_s.start_with?('http://', 'https://')
      true                          # URL — pass through to RubyLLM
    else
      File.exist?(p.to_s)           # String path — check local file
    end
  end
  @auto_selected = false
  @model_selection_reason = nil

  # Derive role from user context for tool access control.
  # user_context is a serialized Hash from the controller with 'is_admin' and 'is_manager' keys.
  @user_role = if @user_context['is_admin']
                 :admin
               elsif @user_context['is_manager']
                 :manager
               else
                 :employee
               end

  # Resolve data domain access from the user's CanCanCan roles.
  # This narrows which views/tables the AI tools can query.
  @account = Account.find_by(id: @user_context['account_id']) if @user_context['account_id']
  @allowed_objects = @account ? Assistant::DataPolicy.(@account) : nil
  @analytics_domains = Array(@user_context['analytics_domains'])

  history_length = @conversation.assistant_messages.count

  # Handle 'auto' model selection
  if model == 'auto' || !MODELS.key?(model)
    selection = self.class.auto_select_model(
      user_message,
      history_length: history_length,
      current_model: @conversation.llm_model_name,
      active_services: Array(@conversation.tool_services)
    )
    @model_key = selection[:model]
    @model_selection_reason = selection[:reason]
    @auto_selected = true
  else
    @model_key = model
  end

  # Complexity-aware upgrade: a session that STARTED cheap but has since
  # revealed its complexity — the model declared a multi-step plan, or the
  # conversation has grown long — climbs the model ladder. Only in auto mode
  # (never override an explicit user pick) and only while the user's monthly
  # budget allows; out of budget → stay on the cheap tier. See
  # Assistant::MonthlyBudget and
  # doc/tasks/202606031730_SUNNY_BUDGET_AND_AUTO_ESCALATION.md.
  #
  # Gate on the stored preference, not @auto_selected: the controller often
  # pre-resolves 'auto' to a concrete key before this point (one classifier
  # pass picks tools + tier), which would otherwise hide auto mode here.
  #
  # opt_in models are exempt, per their contract above. They are absent from
  # LADDER, and LADDER.index returns nil → rung 0, so a declared plan would
  # "upgrade" a deliberately-named model onto a Gemini/Opus rung. The exposed
  # callers are programmatic ones that pass model: while leaving
  # model_preference blank (DailyFocus::ChatRunner): harmless on a fresh
  # conversation, but a re-run reads back the stored execution_plan and would
  # silently move the briefing off the model it was pinned to.
  if auto_model_mode? && !MODELS.dig(@model_key, :opt_in)
    upgrade = Assistant::ComplexityEscalator.upgrade(
      current_model: @model_key,
      plan_step_count: Array(@conversation.execution_plan&.dig('steps')).size,
      history_length: history_length,
      user_context: @user_context
    )
    if upgrade
      @model_key = upgrade[:model]
      @model_selection_reason = upgrade[:reason]
    end
  end

  @model_config = MODELS[@model_key]
end

Instance Attribute Details

#model_key ⇒ String (readonly)

The concrete model key this turn resolved to (e.g. +'claude-sonnet'+). For
an explicit pick this equals the requested model; for +'auto'+ it is the key
the complexity/affinity selector chose. Lets callers base a decision on the
backend that actually ran rather than the requested alias — e.g.
AssistantChatWorker's transient-400 recovery, so an +auto+ turn that
resolved to claude-sonnet retries on a different backend instead of
replaying the model that just 400'd.

Returns:

  • (String) 


665
666
667
 
# File 'app/services/assistant/chat_service.rb', line 665

def model_key
  @model_key
end

Class Method Details

.auto_select_candidate(query, history_length: 0, current_model: nil) ⇒ Hash

Regex/heuristic model selection used when the classifier abstained or was
not run.

Parameters:

  • query (String) —

    the user's message text

  • history_length (Integer) (defaults to: 0) —

    number of prior assistant/user messages

  • current_model (String, nil) (defaults to: nil) —

    model key already in use, if any

Returns:

  • (Hash) —

    { model: String, reason: String, force_switch: Boolean }



440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
 
# File 'app/services/assistant/chat_service.rb', line 440

def self.auto_select_candidate(query, history_length: 0, current_model: nil)
  query_lower = query.downcase.strip
  token_count = estimate_tokens(query)

  is_writing    = query_lower.match?(WRITING_QUERY_PATTERNS)
  is_complex    = query_lower.match?(COMPLEX_QUERY_PATTERNS)
  is_comparison = query_lower.match?(COMPARISON_QUERY_PATTERNS)
  is_compose    = query_lower.match?(COMPOSE_QUERY_PATTERNS)
  is_simple     = query_lower.match?(SIMPLE_QUERY_PATTERNS) && !is_comparison && !is_complex && !is_writing
  long_conversation = history_length > 20

  # Long-form editorial work (blog posts, articles, rewrites) must always run
  # on a Pro/Sonnet-tier model — Flash produces noticeably weaker prose. We
  # force_switch so a conversation that started on Flash doesn't hold writing
  # turns hostage via model affinity. Stay on Sonnet only if the conversation
  # is already on a Claude model; otherwise default to Claude Sonnet.
  if is_writing
    chosen = current_model == WRITING_MODEL_CLAUDE ? WRITING_MODEL_CLAUDE : WRITING_MODEL_DEFAULT
    return { model: chosen, reason: 'Writing/editorial task', force_switch: true }
  end

  # Compose/email tasks are short content generation, not analysis. Keep them
  # on Flash even when the prompt is long (pasted email threads inflate token
  # counts but don't require deep reasoning) — Pro burns most of
  # MAX_TURN_DURATION on extended thinking before any tool runs.
  return { model: 'gemini-flash', reason: 'Compose/email task', force_switch: true } if is_compose && !is_complex

  if is_complex || token_count > 200
    { model: 'gemini-pro', reason: 'Complex analytical query' }
  elsif is_comparison || token_count > 80
    { model: 'gemini-flash', reason: 'Multi-step query' }
  elsif is_simple && !long_conversation
    { model: 'gemini-flash', reason: 'Simple query' }
  else
    { model: 'gemini-flash', reason: long_conversation ? 'Long conversation context' : 'Standard query' }
  end
end

.auto_select_model(query, history_length: 0, current_model: nil, classifier_result: nil, active_services: []) ⇒ Hash

Auto-select the best model based on query complexity.
Works for both analytics and general assistant queries.

Design goals:

  • Prefer the AI classifier's tier when present — it sees the whole prompt
    holistically (multi-task structure, spelling variants, compound asks).
  • Fall back to regex-based candidate selection when the classifier abstained
    or wasn't run (e.g. tests that bypass the LLM call).
  • Default to Gemini Flash for all queries — cheapest option with good quality.
  • Escalate to the Gemini reasoning tier (same gemini-3.7-flash, higher
    thinking-effort budget) only for genuinely complex analytical queries.
  • Claude models are otherwise opt-in (explicit user selection), keeping
    Anthropic costs near zero for auto users — EXCEPT content-authoring
    (blog/email) turns, which always route to Claude Sonnet because the
    Gemini tiers are slow/unreliable on long HTML edits (see
    content_authoring_turn? / WRITING_MODEL_CLAUDE).
  • Model affinity: if the conversation already uses a model, prefer keeping it
    unless the new query demands a higher cost tier. Lateral switches lose
    accumulated reasoning context for no benefit.

Parameters:

  • query (String) —

    The user's question

  • history_length (Integer) (defaults to: 0) —

    Number of messages in conversation history (informational only)

  • current_model (String, nil) (defaults to: nil) —

    Model key currently in use (for affinity)

  • classifier_result (Assistant::QueryClassifier::Result, nil) (defaults to: nil) —

    pre-computed
    classification carrying a model_tier hint. When provided AND its tier is set,
    this overrides the regex candidate.

  • active_services (Array<String>) (defaults to: []) —

    service keys enabled for this conversation
    (tool services + forced chips), used for session-aware routing

Returns:

  • (Hash) —

    { model: 'model-key', reason: 'explanation' }



300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
 
# File 'app/services/assistant/chat_service.rb', line 300

def self.auto_select_model(query, history_length: 0, current_model: nil, classifier_result: nil, active_services: [])
  # Blog editing routes to Opus 5 (1M context) from the first turn — ahead of
  # everything else. These turns are large + tool-heavy and were choking on the
  # cheaper tiers (Gemini 400s #3808, 600s timeouts #4714, truncated-HTML
  # edit loops on large posts — convs 3105/3109). Give them the capable,
  # large-context model up front rather than escalating after failure.
  #
  # Detection is SESSION-aware, not just per-message: a blog session keeps Opus
  # on every turn even when the message itself carries no blog signal (e.g.
  # "retry", "yes confirmed", "clean up the HTML"). Without this, a short
  # continuation in a blog session silently fell back to Flash and thrashed
  # (conv 3117). active_services carries the conversation's enabled tool
  # services + forced chips.
  return { model: BLOG_AUTHORING_MODEL, reason: 'Blog session → Claude Opus 5 (1M context)' } if blog_authoring_turn?(query, classifier_result, active_services, current_model: current_model)

  # Other content-authoring (email/general editorial) turns route to Claude
  # Sonnet, ahead of the classifier/regex candidate AND model affinity. The
  # Gemini is slow/unreliable on long HTML edits — the dropped 3.1-pro-preview
  # snapshot 400'd intermittently (#3808) and burned the full 600s
  # plan-step timeout on complex edits (#4714, conv 3098).
  return { model: WRITING_MODEL_CLAUDE, reason: 'Content-authoring (email/editorial) → Claude' } if content_authoring_turn?(query, classifier_result, active_services, current_model: current_model)

  candidate = if classifier_result&.model_tier
                candidate_from_classifier(classifier_result)
              else
                auto_select_candidate(query, history_length: history_length, current_model: current_model)
              end

  # Some intents (compose/email, writing) are strong enough signals that we
  # override model affinity. Compose pulls down to Flash so a long pasted
  # email thread doesn't burn the whole turn budget on Pro's extended
  # thinking (PR #618 / conv 1233). Writing pushes UP to Pro/Sonnet so we
  # never produce blog content on Flash (conv 1639 / Julia's feedback).
  return candidate.except(:force_switch) if candidate[:force_switch]

  # Affinity never preserves an opt_in model: it exists so auto-selected
  # tiers keep their context continuity, not to trap a session on a
  # preview the user explicitly opted into. Once the dropdown is back on
  # 'auto', routing returns to the standard tiers immediately.
  if current_model.present? && MODELS.key?(current_model) && !MODELS[current_model][:opt_in]
    candidate_tier = COST_TIER_RANK[MODEL_COST_TIER[candidate[:model]]] || 0
    current_tier   = COST_TIER_RANK[MODEL_COST_TIER[current_model]] || 0

    return { model: current_model, reason: "#{candidate[:reason]} (keeping #{current_model} for context continuity)" } if candidate_tier <= current_tier
  end

  candidate
end

.available_models ⇒ Array<Hash>

Returns the list of models available in the UI picker, with Auto first.

Returns:

  • (Array<Hash>) —

    each hash has :key, :label, :cost, and :model_id keys



599
600
601
602
603
604
605
 
# File 'app/services/assistant/chat_service.rb', line 599

def self.available_models
  auto_option = [{ key: 'auto', label: 'Auto (Smart Select)', cost: :auto, model_id: nil }]
  model_options = MODELS.map do |key, config|
    { key: key, label: config[:label], cost: config[:cost], model_id: config[:id] }
  end
  auto_option + model_options
end

.compact_consumed_tool_results!(messages) ⇒ Integer

Shrink already-consumed tool results in a live transcript, in place.

"Consumed" = anything before the LAST assistant message: the model has
already produced a response from those results, so re-sending the raw
payload on every remaining round of the turn buys nothing.

Two 2.0 corrections live here. RubyLLM 2.0 Messages are immutable
(+content+ is attr_reader), so a compacted result REPLACES its slot
instead of being mutated — the 1.x msg.content = raises NoMethodError
on 2.0, inside a before_message callback, which would take down any turn
long enough to trip it. And the shrink goes through truncate_result
rather than first(N): the raw cut landed mid-key/mid-value and then
prefixed plain text, so what the model re-read for the rest of the turn
was unparseable — the exact failure truncate_result exists to prevent,
applied to the copy the model is actively working from.

Shared with PlanOrchestrator#install_step_mid_turn_compaction!, which ran
a byte-identical copy of this loop.

Parameters:

  • messages (Array<RubyLLM::Message>) —

    mutated in place

Returns:

  • (Integer) —

    number of tool results compacted



1774
1775
1776
1777
1778
1779
1780
1781
1782
1783
1784
1785
1786
1787
1788
1789
1790
1791
1792
1793
1794
1795
1796
1797
 
# File 'app/services/assistant/chat_service.rb', line 1774

def self.compact_consumed_tool_results!(messages)
  return 0 if messages.size < MID_TURN_MIN_MESSAGES

  boundary_idx = messages.rindex { |msg| msg.role == :assistant }
  return 0 unless boundary_idx

  compacted = 0
  messages.each_with_index do |msg, idx|
    break if idx >= boundary_idx
    next unless msg.role == :tool

    content_str = msg.content.to_s
    next if content_str.length <= MID_TURN_COMPACT_THRESHOLD
    next if MID_TURN_SKIP_PREFIXES.any? { |prefix| content_str.start_with?(prefix) }

    messages[idx] = RubyLLM::Message.new(
      **msg.to_h,
      content: Assistant::ChatToolBuilder.truncate_result(content_str, max_chars: MID_TURN_KEEP_CHARS)
    )
    compacted += 1
  end

  compacted
end

.estimate_tokens(text) ⇒ Integer

Rough token estimate (1 token ≈ 4 chars for English).
Used only for heuristic model-complexity selection, not billing.

Parameters:

  • text (String) —

    Text to estimate tokens for

Returns:

  • (Integer) —

    Estimated token count



482
483
484
485
486
 
# File 'app/services/assistant/chat_service.rb', line 482

def self.estimate_tokens(text)
  return 0 if text.blank?

  (text.length / 4.0).ceil
end

.label_for_model(model_key) ⇒ String

Resolve a stored model preference / llm_model_name (e.g. 'gemini-pro') to a
human-readable label that includes the actual underlying model id (e.g.
"Gemini 3.5 Flash · Reasoning"). Used by the chat picker and history badges so
users can see WHAT model actually ran a turn — not just the dropdown alias.
Returns the stored value verbatim when no MODELS entry matches.

Parameters:

  • model_key (String, Symbol, nil) —

    the stored model key

Returns:

  • (String) —

    human-readable label or the original key



615
616
617
618
619
620
621
622
 
# File 'app/services/assistant/chat_service.rb', line 615

def self.label_for_model(model_key)
  return 'Auto (Smart Select)' if model_key.to_s == 'auto'

  config = MODELS[model_key.to_s]
  return model_key.to_s if config.nil?

  config[:label]
end

Instance Method Details

#call(&block) ⇒ Result

Execute the chat with streaming response.
Messages auto-persist to assistant_messages via acts_as_chat
with token tracking, tool calls, and thinking traces.

Yields content chunks as they're generated.

Returns:

  • (Result) —

    content and usage stats



673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
 
# File 'app/services/assistant/chat_service.rb', line 673

def call(&block)
  raise ArgumentError, 'Block required for streaming' unless block_given?

  @streamer = block
  @start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
  @full_response = +''

  configure_conversation(reset_advisor_turn: true)
  supersede_unanswered_user_message!

  # Tell the conversation who the actual sender is so AssistantMessage
  # can stamp sender_id on the persisted user message.
  @conversation.current_sender_id = @user_context['party_id']

  # Stream the response — conversation.ask() auto-persists user + assistant messages.
  # The return value of ask() is the fully-assembled StreamAccumulator message with
  # correct input/output token counts (not the last streaming chunk, which has nil tokens).
  streamer_proc = build_streamer_proc

  final_message = with_instrumented_llm_call(feature: 'assistant_chat') do
    with_tool_halt do
      if @attachments.present?
        ask_with_attachments(user_message, @attachments, &streamer_proc)
      else
        @conversation.ask(user_message, &streamer_proc)
      end
    end
  end

  halt_result = handle_halt(final_message, streamer_proc, label: 'call')
  return halt_result if halt_result

  build_result(final_message)
rescue Assistant::Cancelled, RubyLLM::CancelledError
  Rails.logger.info("[Assistant::ChatService] Cancelled by user (call) — conversation #{@conversation.id}")
  build_cancelled_result
rescue RubyLLM::ContextLengthExceededError => e
  Rails.logger.error("[Assistant::ChatService] Context length exceeded: #{e.message}")
  raise
end

#complete_only(&block) ⇒ Result

Retry path after emergency compaction: reconfigure the conversation and
call complete() directly. The user message is already persisted from the
prior attempt — to_llm replays it from DB. Skips ask() to avoid duplicates.

Returns:

  • (Result) —

    content and usage stats



718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
 
# File 'app/services/assistant/chat_service.rb', line 718

def complete_only(&block)
  raise ArgumentError, 'Block required for streaming' unless block_given?

  @streamer = block
  @start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
  @full_response = +''

  configure_conversation(reset_advisor_turn: false)
  @conversation.current_sender_id = @user_context['party_id']

  streamer_proc = build_streamer_proc

  final_message = with_instrumented_llm_call(feature: 'assistant_chat') do
    with_tool_halt do
      llm_chat = @conversation.to_llm
      # Recovery replays (body-less 400 #3808, blank-after-tools escalation,
      # emergency compaction) rebuild history from the DB, which stores the
      # user message text-only. Re-apply the attachments the worker still has
      # on disk so a vision turn that retries on another model doesn't answer
      # the image question blind.
      apply_attachments_to_llm_chat!(llm_chat, user_message, @attachments) if @attachments.present?
      llm_chat.complete(&streamer_proc)
    end
  end

  halt_result = handle_halt(final_message, streamer_proc, label: 'complete_only')
  return halt_result if halt_result

  build_result(final_message)
rescue Assistant::Cancelled, RubyLLM::CancelledError
  Rails.logger.info("[Assistant::ChatService] Cancelled by user (complete_only) — conversation #{@conversation.id}")
  build_cancelled_result
end

#emit_status(message) ⇒ void (protected)

This method returns an undefined value.

Emit a status update for the UI (non-content, just progress indicator).
Also used by Assistant::PlanOrchestrator (via Object#send).

Parameters:

  • message (Symbol, Hash) —

    status event; a Hash carries extra context
    such as the tool name and arguments



1714
1715
1716
 
# File 'app/services/assistant/chat_service.rb', line 1714

def emit_status(message)
  @on_status&.call(message)
end

#stream_content(content) ⇒ void (protected)

This method returns an undefined value.

Stream content to client AND capture for conversation history.
Also used by Assistant::PlanOrchestrator (via Object#send).

Parameters:

  • content (String) —

    the content chunk to stream



1560
1561
1562
1563
 
# File 'app/services/assistant/chat_service.rb', line 1560

def stream_content(content)
  @full_response << content
  streamer.call(content)
end

#with_instrumented_llm_call(feature:, source: 'sunny') { ... } ⇒ Object (protected)

Wraps an LLM call with PaperTrail audit context, CurrentScope user, instrumentation
metadata, and transient network retries. Every LLM round (ask, complete, agent.ask)
should go through this so audit trail, cost logging, and retries are consistent.
Also used by Assistant::PlanOrchestrator (via Object#send). On a body-less
RubyLLM::BadRequestError it attaches the outgoing request shape to AppSignal
(#3808) before re-raising.

Parameters:

  • feature (String) —

    instrumentation feature tag (e.g. 'assistant_chat')

  • source (String) (defaults to: 'sunny') —

    PaperTrail controller-info source (default 'sunny')

Yields:

  • the LLM call to instrument and retry

Returns:

  • (Object) —

    the yielded block's return value (e.g. the final RubyLLM message)

Raises:

  • (RubyLLM::BadRequestError) —

    re-raised after diagnostics are attached



819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
 
# File 'app/services/assistant/chat_service.rb', line 819

def with_instrumented_llm_call(feature:, source: 'sunny', &)
  sender_id = @user_context['party_id'] || @conversation.user_id
  whodunnit = sender_id.to_s.presence || 'Sunny'
   = Account.where(party_id: sender_id).pick(:id)

  # Multi-round agentic turns: the global instrumentation audience_member only
  # sees the FINAL tool-loop round's usage (RubyLLM returns the final
  # response up every recursive complete()), undercounting Sunny chat ~2x —
  # turns average ~18 billed rounds. assistant_chat is excluded from that
  # audience_member (MANUALLY_LOGGED_FEATURES); instead we sum this turn's
  # per-round assistant_messages in log_turn_usage! below, which reconcile
  # to the Anthropic Cost API within ~10%.
  sum_turn = feature == 'assistant_chat'
  # The ledger's OWN high-water mark, not the message table's. RubyLLM
  # creates a usage row when the attempt starts and only links it to a
  # message afterwards (chat_methods.rb persist_usage_entry →
  # link_usage_entries), so an attempt that fails before its message is
  # persisted keeps message_id NULL forever. Bounding the turn by message id
  # would silently drop exactly those rows — the billed failures this rollup
  # exists to stop losing.
  since_usage_id = sum_turn ? @conversation.ruby_llm_usages.maximum(:id).to_i : nil

  result = PaperTrail.request(
    whodunnit: whodunnit,
    controller_info: {
      source: source,
      sender_id: sender_id,
      sender_name: @user_context['full_name'],
      conversation_id: @conversation.id,
      conversation_url: "/en-US/assistant/#{@conversation.id}"
    }
  ) do
    CurrentScope.with_user_id(sender_id) do
      RubyLLM::Instrumentation.with(
        **(@log_metadata || {}),
        feature: feature,
        conversation_id: @conversation.id,
        log_subject: @conversation,
        log_account_id: 
      ) do
        with_llm_network_retries(&)
      end
    end
  end

  result
rescue RubyLLM::BadRequestError => e
  # A 400 on a STREAMING turn arrives body-less, so RubyLLM surfaces the
  # generic "Invalid request - please check your input" with no provider
  # detail — which left AppSignal #3808 undiagnosable for months (the real
  # reason is in the REQUEST we sent, not the empty response). Snapshot the
  # outgoing request shape onto the AppSignal transaction so the NEXT
  # occurrence names the offending payload (after #1069 fixed the dominant
  # Opus-4.7+-temperature cause, any residual cause is otherwise opaque).
  # Diagnostics must never mask the real error — re-raise unconditionally.
  attach_llm_request_diagnostics(e)
  raise
ensure
  # In `ensure`, not on the success path: a raised error or a user
  # cancellation unwinds past the normal return, and those attempts are
  # exactly the ones that consumed tokens without producing a message.
  # Logging only on success threw away the spend this rollup exists to
  # capture — the same loss the message_id filter used to cause one layer
  # down. log_turn_usage! swallows its own exceptions, so it cannot replace
  # the in-flight error.
  #
  # since_usage_id must be non-nil, not merely sum_turn: if the high-water
  # query itself raised, the boundary was never captured and a nil would
  # scope `id >= 1` — re-logging the conversation's entire history.
  log_turn_usage!(since_usage_id, ) if sum_turn && since_usage_id
end