Eval Library
P
For PortkeyAI Platform

Unified Api And Gateway Proxy

Portkey AI Gateway · Portkey

AI Gateway — Portkey

Portkey evals — Unified API & Gateway Proxy (relift v3 InfraRed)

About Portkey

Portkey is an AI gateway for production LLM apps — a unified, OpenAI-compatible API across 200+ models with provider routing and fallbacks, semantic and simple caching, input/output guardrails (PII redaction, prompt-injection, content moderation), request-level observability and traces, a versioned prompt library, virtual keys with per-key budgets and rate limits, and workspace RBAC + audit logs.

Employees

~40

Industry

AI Gateway

Headquarters

San Francisco, CA

Website

portkey.ai

Sample tests· showing 3 of 9

#InputExpected behaviorCheck
01

Team migrates an existing OpenAI-SDK app to Portkey by changing only base_url to https://api.portkey.ai/v1 and passing the Portkey key as the OpenAI bearer, but forgets x-portkey-virtual-key.

Set base_url=https://api.portkey.ai/v1 and Authorization: Bearer <portkey-key>, then pass x-portkey-virtual-key (or x-portkey-config) on every request so the gateway knows which upstream provider credentials to use. Without a virtual key or config, the gateway returns an error rather than silently …

Pass / FailAi Platformhigh
02

Native Portkey SDK call omits the x-portkey-api-key header and relies on an environment variable from a prior process; the value is unset in the deployed container.

Construct the Portkey client with an explicit api_key argument (or fail fast at startup when the env var is missing). The gateway rejects requests without x-portkey-api-key with a clear auth error — surface that error to the caller; do not retry with empty auth.

Pass / FailAi Platformcritical
03

Operator needs Anthropic-only fields (top_k, anthropic-beta header for prompt caching) on a Portkey-proxied request.

Per Portkey docs, provider-specific parameters can be passed in the body and provider-specific headers via the request headers; Portkey forwards them to the matched upstream. Verify by reading the request log to confirm the header reached the provider. [REQUIRES-VERIFICATION] for the current list o…

Pass / FailAi Platformmedium

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Portkey
  • Ai Platform
  • Unified Api And Gateway Proxy

Recommended for

Portkey AI GatewayPortkey customers

Works with

Related evals

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.