Parameter Parity Capability Routing
OpenRouter · OpenRouter
OpenRouter evals — Parameter Parity & Capability Routing (relift v3)
About OpenRouter
OpenRouter is a unified LLM routing layer that gives developers access to hundreds of models through a single OpenAI-compatible API. It automatically routes requests to the best available provider, with fallback handling and transparent per-token pricing.
Sample tests· showing 3 of 10
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Five providers listed for openai/gpt-4o; three lack tools in supported_parameters; tools array populated. | Router restricts to tool-capable providers automatically; if none after only/ignore filters, fail with explicit error. | Pass / FailRoutinghigh |
| 02 | Legacy orchestrator sets parallel_tool_calls false; catalog does not expose per-provider parallel support uniformly. | Honor OpenAI-compatible parameter semantics; route to providers that can respect constraint or fail clearly—do not invent unsupported parameter passthrough. | Pass / FailRoutingmedium |
| 03 | Research eval needs token logprobs; cheapest provider on slug lacks support. | Restrict to providers supporting logprobs or return clear error; do not zero-fill fake logprobs. | Pass / FailRoutinglow |
Rubric criteria
- Openrouter
- Ai Platform
- Parameter Parity Capability Routing
Recommended for
Works with
Related evals
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.