Module: Assistant::PromptCache

Defined in:
app/services/assistant/prompt_cache.rb

Overview

Central configuration for Sunny's Anthropic prompt-cache breakpoints.

Sunny's request carries up to three +cache_control+ breakpoints (Anthropic
allows four): the frozen system prefix
(Assistant::PromptComposer#cacheable_system_prompt?), the tool array
(ChatToolBuilder.tools_for_services), and — added by the
+config/initializers/ruby_llm_prompt_caching.rb+ provider patch — the LAST
conversation message.

Two cost levers are configured here:

  • Lever 1 — extended TTL. Breakpoints default to a 1-hour cache TTL
    (PromptCache.ttl == +'1h'+) instead of Anthropic's 5-minute default, so the cached
    prefix survives between a rep's sporadic turns. The 1-hour TTL is GA — it
    needs no beta header, only +cache_control: { type: 'ephemeral', ttl: '1h' }+.
    All breakpoints share one TTL on purpose: Anthropic 400s if a shorter-TTL
    block precedes a longer-TTL one, so a uniform TTL can never trip that.
  • Lever 2 — last-message breakpoint. PromptCache.message_caching_enabled? gates the
    provider patch that caches the conversation history plus the tool results
    that accumulate across the ~18-round tool loop (re-sent at full price on
    every round otherwise).
  • Lever 3 — mid-conversation system message. PromptCache.mid_conversation_system_enabled?
    gates relocating the volatile system tail (active plan, planning mandate)
    out of the top-level +system+ field and into a +{ role: 'system' }+ entry
    appended to +messages+. Lever 2 is worthless without it: +system+ renders
    BEFORE +messages+ in the hashed prefix, so a volatile tail that changes
    every turn busts the message cache Lever 2 just paid to write.

Each lever has an independent env kill switch for deploy-free rollback.

Class Method Summary collapse

Class Method Details

.cache_control(string_keys: false) ⇒ Hash

The +cache_control+ hash for one breakpoint, in the key style the call site
needs: system-prompt Content blocks use symbol keys; tool +provider_options+
use string keys. Both serialize to identical JSON.

Parameters:

  • string_keys (Boolean) (defaults to: false) —

    return +'type'+/+'ttl'+ string keys when true

Returns:

  • (Hash) —

    e.g. +{ type: 'ephemeral', ttl: '1h' }+



70
71
72
73
74
75
76
 
# File 'app/services/assistant/prompt_cache.rb', line 70

def cache_control(string_keys: false)
  type_key = string_keys ? 'type' : :type
  ttl_key  = string_keys ? 'ttl'  : :ttl
  control  = { type_key => 'ephemeral' }
  control[ttl_key] = ttl if ttl
  control
end

.message_caching_enabled? ⇒ Boolean

Whether to stamp a +cache_control+ breakpoint on the last conversation
message (Lever 2). Defaults on; set +SUNNY_DISABLE_PROMPT_CACHE_MESSAGES+
to disable it without touching the always-on system + tool breakpoints.

Returns:

  • (Boolean) 


41
42
43
 
# File 'app/services/assistant/prompt_cache.rb', line 41

def message_caching_enabled?
  !ENV['SUNNY_DISABLE_PROMPT_CACHE_MESSAGES'].to_b
end

.mid_conversation_system_enabled? ⇒ Boolean

Whether to relocate the volatile system tail into a mid-conversation
+{ role: 'system' }+ message (Lever 3, see the provider patch). Defaults
on; set +SUNNY_DISABLE_MID_CONVERSATION_SYSTEM+ to fall back to shipping
the tail as a second top-level system block.

Returns:

  • (Boolean) 


51
52
53
 
# File 'app/services/assistant/prompt_cache.rb', line 51

def mid_conversation_system_enabled?
  !ENV['SUNNY_DISABLE_MID_CONVERSATION_SYSTEM'].to_b
end

.ttl ⇒ String?

The cache TTL applied to every breakpoint. +'1h'+ (the default) extends the
cache lifetime to one hour; any other value (e.g. +SUNNY_PROMPT_CACHE_TTL=5m+)
falls back to Anthropic's 5-minute default and stops sending the beta header.

Returns:

  • (String, nil) —

    +'1h'+ for the extended TTL, else nil for the default



60
61
62
 
# File 'app/services/assistant/prompt_cache.rb', line 60

def ttl
  ENV.fetch('SUNNY_PROMPT_CACHE_TTL', '1h') == '1h' ? '1h' : nil
end