
Provider Routing And Fallbacks
Portkey AI Gateway · Portkey
AI Gateway — Portkey
Evaluates Portkey's Provider Routing & Fallbacks across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Gateway eval coverage.
About Portkey
Portkey is an AI gateway for production LLM apps — a unified, OpenAI-compatible API across 200+ models with provider routing and fallbacks, semantic and simple caching, input/output guardrails (PII redaction, prompt-injection, content moderation), request-level observability and traces, a versioned prompt library, virtual keys with per-key budgets and rate limits, and workspace RBAC + audit logs.
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Config strategy={mode:'fallback'} with targets=[anthropic_vk, openai_vk, bedrock_vk]; primary is Anthropic, fallbacks fire on listed status codes. | On primary upstream failure matching on_status_codes, the gateway tries the next target in declared order. Verify the cascade by reading x-portkey-provider on the response and the per-target attempt log. Order of targets[] is the contract — do not reorder between deploys without intent. | Pass / FailAi Platformcritical |
| 02 | Config strategy={mode:'loadbalance'} with targets weighted 70/20/10 across three virtual keys. | Weights are relative; gateway distributes traffic across targets in expectation matching the weights. Verify by aggregating x-portkey-provider over a representative sample. Do not expect per-request determinism — loadbalance is stochastic. | Pass / FailAi Platformhigh |
| 03 | Config strategy={mode:'conditional'} with query rules routing requests with metadata.env='prod' to a paid virtual key and metadata.env='dev' to a sandbox virtual key. | Send x-portkey-metadata JSON with env on every request. Conditional strategy evaluates query rules in declared order and routes to the matched target; a default target must be set for rule-miss. Verify with trace logs that prod metadata reaches the prod target. | Pass / FailAi Platformhigh |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Portkey
- Ai Platform
- Provider Routing And Fallbacks
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Provider Routing And Fallbacks eval for Portkey Portkey AI Gateway test?+
Evaluates Portkey's Provider Routing & Fallbacks across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Gateway eval coverage.
How is the Provider Routing And Fallbacks eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Provider Routing And Fallbacks pack for Portkey Portkey AI Gateway contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Provider Routing And Fallbacks pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.