Byok Isolation Usage Provenance
OpenRouter · OpenRouter
OpenRouter evals — BYOK Isolation & Usage Provenance (relift v3)
About OpenRouter
OpenRouter is a unified LLM routing layer that gives developers access to hundreds of models through a single OpenAI-compatible API. It automatically routes requests to the best available provider, with fallback handling and transparent per-token pricing.
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Tenant uploaded Anthropic key; response usage shows is_byok true and cost near zero on OpenRouter ledger while upstream_inference_cost populated in cost_details. | Map is_byok true to pass-through upstream billing for tenant key spend while separating OpenRouter platform fees per documented cost_details fields. | Pass / FailBillingcritical |
| 02 | Dashboard shows active Anthropic BYOK; application still sends OPENROUTER_API_KEY only; expects tenant key consumption. | OpenRouter prioritizes tenant BYOK for matching provider before platform pooled keys, reflected in is_byok and provenance metadata when enabled. | Pass / FailRoutinghigh |
| 03 | MSP hosts multiple customers under one org; each customer workspace registers separate OpenAI BYOK keys. | Keys are workspace-scoped; routing and usage attribution never bleed across workspaces even within shared org billing. | Pass / FailSecuritycritical |
Rubric criteria
- Openrouter
- Ai Platform
- Byok Isolation Usage Provenance
Recommended for
Works with
Related evals
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.