Router Metadata Observability
OpenRouter · OpenRouter
OpenRouter evals — Router Metadata & Observability (relift v3)
About OpenRouter
OpenRouter is a unified LLM routing layer that gives developers access to hundreds of models through a single OpenAI-compatible API. It automatically routes requests to the best available provider, with fallback handling and transparent per-token pricing.
Sample tests· showing 3 of 10
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | SRE enables pipeline tracing without storing prompts; needs attempt-level visibility. | Set X-OpenRouter-Experimental-Metadata on request per router metadata guide; parse openrouter_metadata from response header/body as documented. | Pass / FailObservabilityhigh |
| 02 | Identical prompts produce fast responses; SRE sees empty metadata header intermittently. | Router metadata guide documents stripping on cache hits; use generation id lookup or bypass cache for full pipeline when debugging. | Pass / FailObservabilitymedium |
| 03 | Central logging pipeline stores model, generation id, is_byok, cost, and metadata attempts when present. | Log model slug, X-Generation-Id, usage.cost, usage.is_byok, provider winner from metadata, HTTP status, without storing full prompts in clear text unless policy allows. | Pass / FailObservabilitylow |
Rubric criteria
- Openrouter
- Ai Platform
- Router Metadata Observability
Recommended for
Works with
Related evals
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.