Model Catalog Alias Stability
OpenRouter · OpenRouter
OpenRouter evals — Model Catalog & Alias Stability (relift v3)
About OpenRouter
OpenRouter is a unified LLM routing layer that gives developers access to hundreds of models through a single OpenAI-compatible API. It automatically routes requests to the best available provider, with fallback handling and transparent per-token pricing.
Sample tests· showing 3 of 10
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Cron caches models JSON for routing agent; new provider added mid-day for meta-llama slug. | Recommend periodic refresh with versioned ETag or timestamped cache invalidation; stale cache risks require_parameters mismatch errors at runtime. | Pass / FailCatalogmedium |
| 02 | Config uses anthropic/claude-3.5-sonnet:beta style moving target; eval harness expects fixed tokenizer limits. | Warn that ~latest and similar aliases can retarget underlying weights without slug change notification; pin explicit dated slugs for regression-sensitive workloads. | Pass / FailCataloghigh |
| 03 | Legacy config uses OpenAI/GPT-4o mixed case; OpenRouter expects vendor/slug lowercase paths. | Treat slugs as case-sensitive per catalog exact match; normalize using GET /api/v1/models canonical id before deploy. | Pass / FailCataloglow |
Rubric criteria
- Openrouter
- Ai Platform
- Model Catalog Alias Stability
Recommended for
Works with
Related evals
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.