Configs And Virtual Keys
Portkey AI Gateway · Portkey
AI Gateway — Portkey
Portkey evals — Configs & Virtual Keys (relift v3 InfraRed)
About Portkey
Portkey is an AI gateway for production LLM apps — a unified, OpenAI-compatible API across 200+ models with provider routing and fallbacks, semantic and simple caching, input/output guardrails (PII redaction, prompt-injection, content moderation), request-level observability and traces, a versioned prompt library, virtual keys with per-key budgets and rate limits, and workspace RBAC + audit logs.
Sample tests· showing 3 of 10
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Team has OpenAI, Anthropic, Bedrock credentials; wants to rotate the OpenAI key without code changes. | Create a virtual key per upstream credential set; clients pass x-portkey-virtual-key (or it's set in a config) and never see the raw provider credential. Rotation = re-issue the underlying credential in the virtual key — operator code is unchanged. | Pass / FailAi Platformcritical |
| 02 | Virtual key configured 60 requests/minute; bursty caller sends 100 in 10 seconds. | Gateway rejects requests above the limit within the window (typical 429 response with retry semantics) [REQUIRES-VERIFICATION for the exact response shape]. Operator implements client-side backoff aligned with the limit; do not retry blindly. | Pass / FailAi Platformhigh |
| 03 | Virtual key has a 'monthly' budget; question is the rollover anchor (calendar month vs 30-day rolling). | Confirm the budget rollover semantics in the dashboard and pin them in operator runbooks [REQUIRES-VERIFICATION for the documented anchor and timezone]. Do not assume calendar-month rollover — surprise budget resets in the middle of a billing cycle break finance reconciliation. | Pass / FailAi Platformmedium |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Portkey
- Ai Platform
- Configs And Virtual Keys
Recommended for
Works with
Related evals
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.