OpenRouter — capability study and how Heatwave uses it

Studied 2026-08-11 against the live docs (https://openrouter.ai/docs/llms.txt)
while evaluating open-weight models (GLM, Qwen, Kimi) for the Daily Focus
briefing workload. OpenRouter is wired at the RubyLLM level
(config.openrouter_api_key); a model is callable from app code once it has
Assistant::ChatService::MODELS + AiModelConstants entries — the opt-in
test fleet (glm-5.2, qwen3.7-plus, kimi-k2.6) is registered that way.

The one-paragraph mental model

One OpenAI-shaped endpoint (POST /api/v1/chat/completions, also /messages
Anthropic-shape and /responses) in front of ~500 models across ~60 hosting
providers. Requests carry OpenRouter-only extensions: models (fallback
array), provider (routing preferences), session_id, service_tier,
plugins. Responses carry exact billing (usage.cost, native-tokenizer token
counts, cache details) on every request — no polling needed. Everything else
below is layered on that pipe.

Routing

  • Provider routing (provider: {...}): order, only, ignore
    (provider allow/deny), allow_fallbacks, require_parameters (route only
    to hosts supporting every param you send — set this when sending
    tools    , otherwise a host that ignores tools is eligible),
    quantizations (fp8/bf16… filter — open-weight hosts differ!), sort
    (price/throughput/latency), max_price (hard $/Mtok ceiling),
    data_collection: 'deny', zdr: true.
  • Suffix shorthands: :nitro = sort throughput, :floor = sort price.
    Variants also exist as :free, :extended, :thinking, :online.
  • Auto Exacto — on by default for every request that includes tools.
    Replaces price-weighted routing with quality-first provider ordering built
    from live tool-call schema validation error rates (InvalidJson /
    UnknownName / SchemaMismatch per request), throughput, and a rolling
    benchmark harness (GPQA Diamond + Tau2-Bench Airline). Setting
    sort: 'price' or :floor opts out — don't, on agentic workloads.
  • Model fallbacks (models: [...]): ordered array tried on any error
    (rate limit, downtime, context overflow, moderation). Billed at whichever
    model actually served. The /messages shape calls it fallbacks
    (max 3, model-only entries). model wins over models[0] — verified
    live in both directions (model: glm, models: [qwen] served GLM;
    model: qwen, models: [glm] served Qwen), so the array is a true fallback
    list and never a router that quietly demotes the model you asked for.
    response.model names whoever actually served, which is what RubyLLM
    records on the usage ledger (model: response.model || @model.id) — so a
    server-side failover is attributed and priced as the model that ran, not
    the one we requested.
  • Auto router (openrouter/auto): a lightweight classifier assigns each
    prompt one of ~30 task types, then ranks candidates by real community spend
    over a trailing 7-day window; cost_tier picks a price band. Billed at
    the chosen model's rate. Inspect decisions via X-OpenRouter-Metadata: enabled. Not for the briefing (we need a pinned model per test), but fine
    for low-stakes internal one-offs.

Caching — two distinct layers

  • Prompt caching (provider-side prefix cache): automatic on OpenAI, Grok,
    Groq, DeepSeek, Moonshot, Z.AI (GLM); explicit Anthropic-style
    cache_control breakpoints on Anthropic, Qwen (5-min TTL, writes
    1.25×, reads 0.1×), Gemini. Read discounts: DeepSeek/Qwen 0.1×, Z.AI
    ~0.2×, Moonshot/Gemini/Grok 0.25×. OpenRouter translates Anthropic-style
    markers to the target provider's dialect. Sticky routing: after a cache
    hit, follow-up requests pin to the same host for 5 min; pass a
    session_id (body or x-session-id) to pin immediately — this is why
    multi-step plan runs (briefings) should always send one. Cache telemetry
    comes back in usage.prompt_tokens_details (cached_tokens,
    cache_write_tokens).
  • Response caching (OpenRouter-side, whole-response): SHA-256 of the
    normalized body per API key; identical request → free replay (0 tokens
    billed, own generation id). Off by default; enable per request with
    X-OpenRouter-Cache: true (+ X-OpenRouter-Cache-TTL, default 300s, max
    24h). Useless for briefings (every prompt unique), handy for repeated
    identical utility calls. Disabled entirely under account-level ZDR.

Privacy — the part that matters for briefings

Briefing prompts carry customer PII, and open-weight models route to
third-party hosts with wildly different retention policies (unverifiable
policies are assumed worst-case: retains + trains).

  • provider: {data_collection: 'deny'} — exclude hosts that may train on or
    collect prompts. Filters on the host's own declared policy; unverifiable
    policies are assumed worst-case.
  • provider: {zdr: true} — restrict to Zero-Data-Retention endpoints
    (GET /api/v1/endpoints/zdr, public, no auth). Same commitment, but as an
    attested, enumerable list rather than a self-declaration. Implicit prompt
    caching stays eligible (in-memory ≠ retention).

Both are on for every Heatwave OpenRouter call (decided 2026-08-12). ZDR is
the one that binds; data_collection stays because it costs nothing and the
two filter on different evidence. Measured the day we adopted it, counting only
hosts that also declare tool_choice (we send require_parameters):

Model Endpoints ZDR + tool_choice
z-ai/glm-5.2 (briefing) 32 21
moonshotai/kimi-k2.6 21 15
qwen/qwen3.7-plus 1 0

The cheapest ZDR GLM endpoint bills $0.50/$3.15 — the same floor we already
price against — so the briefing lost nothing. Qwen did: it publishes a single
endpoint and it isn't zero-retention, which is why the glm-5.2 models array
fails over to Kimi rather than to the A/B runner-up. Re-check the list before
adding an OpenRouter model — hosts move on and off it, and an all-non-ZDR
model is simply unroutable for us now.

  • OpenRouter itself does not retain prompts unless prompt logging is opted
    into. Account-wide ZDR toggles exist per model group; per-request zdr
    can only add enforcement, never disable account settings.
  • Guardrails (account feature) can enforce ZDR per API key and add
    sensitive-info detection / prompt-injection regex screens.
  • EU-only routing exists (eu.openrouter.ai, enterprise, by request).

Billing & observability

  • Exact cost on every response: usage.cost (credits actually charged),
    cost_details.upstream_inference_cost, cached/reasoning token splits,
    native-tokenizer counts. Beats static pricing tables — when we productize
    an OpenRouter model, Ai::CostCalculator should prefer response-reported
    cost over its own arithmetic (no pricing-table drift). Post-hoc:
    GET /api/v1/generation?id=....
  • Zero completion insurance: empty/errored completions aren't billed.
  • Workspaces + budgets: per-workspace spending caps with automatic
    enforcement — the platform-side twin of our ai_cost guards.
  • Custom classifiers (workspace feature, dashboard-configured): a cheap
    model asynchronously tags every generation against a taxonomy you define
    (≤8 dimensions — department, task type, complexity presets); zero request
    latency, results in Activity log rollups. We already attribute internally
    via ai_usage_logs.feature, so this is redundant for us unless we want
    OpenRouter-side rollups without shipping metadata.
  • Broadcast: zero-code trace export (prompts, completions, tokens, cost,
    latency, tool usage) to Datadog, ClickHouse, OpenTelemetry Collector,
    Langfuse, S3, webhooks…; per-destination sampling and a Privacy Mode that
    strips prompt/completion bodies. Relevant to the AppSignal→HyperDX
    migration: HyperDX ingests OTLP, so OpenRouter traffic could stream into
    it with no app-side instrumentation. Free (no OpenRouter fee documented).

Batch API (beta)

POST /api/beta/batches — inline JSON array of requests (no JSONL upload;
endpoint and model fields must serialize before requests), 24h
window, results inline on poll, ~50% off token pricing, text-only, any
model. Single-shot only — no tool loops, so it cannot run the briefing
pipeline; it fits bulk translations, classification/backfill jobs, and
embeddings — anywhere we currently loop a worker over rows calling a model
once per row.

Everything else, one line each

  • Presets (@preset/slug): server-side named configs (model, provider
    rules, system prompt, tools), versioned, shallow-merged under the request.
    We keep model config in code/Setting — adopting presets would split the
    source of truth; skip.
  • Service tiers (service_tier: flex|priority): flex ≈ 50% off at
    higher latency — OpenAI/Google/xAI models only, so irrelevant to the
    open-weight tests; flex never falls back to standard (capacity errors
    surface).
  • Server tools: OpenRouter-operated web search/fetch, shell,
    apply-patch, subagents. We have our own tool layer; unused.
  • Plugins: web, file-parser (PDF), response-healing (repairs
    truncated/malformed JSON output), context-compression. Response-healing
    is worth remembering for structured-output jobs on small models.
  • Structured outputs: response_format: json_schema with strict —
    pair with provider.require_parameters so only schema-capable hosts are
    eligible.
  • Multimodal/media APIs: image/video/TTS/STT generation endpoints exist;
    out of scope here.
  • Embeddings: /v1/embeddings shape supported (incl. in batch).
  • API-key management API + OAuth PKCE: programmatic key provisioning —
    useful if we ever hand per-agent scoped keys.
  • user field: stable end-user id for abuse detection and trace
    grouping. Unused here — populating it puts an internal identifier in a body
    that routes to third-party hosts, so it needs the same privacy call as any
    other PII in the prompt, not a default-on.

How this maps onto RubyLLM (our stack)

RubyLLM::Chat exposes everything needed — no gem changes:

OpenRouter feature RubyLLM call
Body extensions (provider, models, session_id, service_tier) with_provider_options(...)
Headers (X-OpenRouter-Cache*, x-session-id, X-OpenRouter-Metadata) with_headers(...)
Prompt-cache breakpoints with_caching (provider module renders cache_control)
Client-side fallback chain with_fallbacks (or server-side via models in provider options)

The provider module (ruby_llm/providers/openrouter/chat.rb) already maps
reasoning params and strips strict from schemas where hosts reject it.

with_provider_options REPLACES, it does not merge (@provider_options = provider_options.to_h). Anything set later in a configuration sequence — output
limits, generationConfig — wipes the routing prefs set earlier, and the
request still succeeds, just unrouted and unpinned. Read the current hash back
off the chat and merge (Assistant::ChatService#merge_provider_options!); a
test that exercises one setter against a fresh chat cannot see this.

Two failure layers, and why both

models: (above) fails over between models inside OpenRouter — same request,
no extra round trip, but useless when OpenRouter itself is the thing that's
down. RubyLLM::Chat#with_fallbacks is the outer layer: it retries the turn on
a different provider entirely, on the transient error classes in
RubyLLM::Fallback::DEFAULT_ERRORS. A fallback swaps model, provider and
connection but keeps provider_options, so the OpenRouter body extensions
have to be cleared in a before_fallback hook or the non-OpenRouter provider
400s on provider / models / session_id.

require_parameters narrows the host pool — count before adding a param

Every parameter we send is a filter under require_parameters: true, and hosts
declare support unevenly. Measured Aug 2026 across the endpoints for our three
models: tool_choice is declared by GLM 31/32, Qwen 1/1, Kimi 20/21 — a cheap
filter worth keeping. parallel_tool_calls is declared by GLM 1/32, Qwen 0/1,
Kimi 1/21, so sending it emptied the pool outright and every request failed with
No endpoints found that can handle the requested parameters. That error names
the parameters, not the count, which reads like a credentials or privacy problem
— check parameter declarations before assuming either.

chat.with_provider_options(
  provider: {
    data_collection: 'deny',      # PII never lands on a training host
    zdr: true,                    # attested zero-retention endpoints only
    require_parameters: true      # tools/schema support is mandatory, not best-effort
  },
  session_id: "conv-#{conversation.id}"  # sticky routing → cache hits across plan steps
)

Leave sort unset (preserves Auto Exacto's quality-first ordering on tool
requests). Record usage.cost when available instead of recomputing from
pricing tables.

Enforcement lives on the key, not the request

The request-level zdr flag ORs with account and guardrail settings — it can
only add enforcement, never relax it — so the durable control is a
Guardrail bound to the API key: a key that cannot route to a retaining host
regardless of what any caller sends. Keep sending the body flag anyway; it
still holds if a key is ever provisioned without the guardrail, and it states
the requirement at the call site.

Provisioned 2026-08-12 via the management API (management_api_key in
credentials; every call below needs it, not the inference key):

# the guardrail: all five ZDR model groups on, no budget fields
curl -X POST https://openrouter.ai/api/v1/guardrails \
  -H "Authorization: Bearer $MANAGEMENT_KEY" -H 'Content-Type: application/json' \
  -d '{"name":"Heatwave ZDR","workspace_id":"…","enforce_zdr_anthropic":true,
       "enforce_zdr_openai":true,"enforce_zdr_google":true,
       "enforce_zdr_xai":true,"enforce_zdr_other":true}'

# bind it to the key (note: snake_case body, camelCase in the SDK docs)
curl -X POST https://openrouter.ai/api/v1/guardrails/<id>/assignments/keys \
  -H "Authorization: Bearer $MANAGEMENT_KEY" -H 'Content-Type: application/json' \
  -d '{"key_hashes":["<key hash from GET /api/v1/keys>"]}'

Live state: guardrail f7a69395-870d-46d4-8663-181164d5b6fa, assigned to key
heatwave-production. enforce_zdr_other is the group that governs GLM/Kimi;
the four frontier groups are on as free insurance since we don't route
Anthropic/OpenAI/Google/xAI through OpenRouter at all.

Three things that cost time and will again:

  • The key is in Default Workspace, not the heatwave workspace, despite
    credentials.openrouter.workspace reading heatwave. A guardrail only binds
    to keys in its own workspace — assignment 400s with "Some keys do not belong
    to the same workspace" otherwise. Check GET /api/v1/keys → workspace_id
    before creating one.
  • Assignment lives at /guardrails/<id>/assignments/keys, not
    /guardrails/<id>/keys (404). The flat "all assignments" listing is
    /guardrails/assignments/keys.
  • GET /api/v1/keys keeps reporting guardrail_id: null even when bound.
    The assignment listing is the source of truth, not that field.

Verify enforcement by asking for a model whose endpoints are all non-ZDR —
qwen/qwen3.7-plus is the live example, and it now returns 404 "No endpoints
available matching your guardrail restrictions and data policy" on our key
while z-ai/glm-5.2 serves normally. Budgets were deliberately left null:
a spend cap on this guardrail would silently kill the morning briefing.

Not covered by ZDR: plugins and OpenRouter-operated server tools (web
search, file-parser). Enforcement applies to inference provider routing only.
We use neither, and adopting one would need its own retention review.