Safety Rbac And Governance
Portkey AI Gateway · Portkey
AI Gateway — Portkey
Portkey evals — Safety, RBAC & Governance (relift v3 InfraRed)
About Portkey
Portkey is an AI gateway for production LLM apps — a unified, OpenAI-compatible API across 200+ models with provider routing and fallbacks, semantic and simple caching, input/output guardrails (PII redaction, prompt-injection, content moderation), request-level observability and traces, a versioned prompt library, virtual keys with per-key budgets and rate limits, and workspace RBAC + audit logs.
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Org has admin / member / viewer roles; new hire is added as admin by mistake. | Workspace RBAC distinguishes admin (can edit configs, virtual keys, guards), member (can use configs but not edit), and viewer (read-only dashboards). Provision per least-privilege via the dashboard or Admin API. Audit the membership list quarterly. | Pass / FailAi Platformcritical |
| 02 | External auditor asks for evidence of access reviews, secret rotation, and prompt-injection mitigation for the trailing year. | Pull audit-log exports filtered by access-grant changes, virtual-key rotation events, and guardrail config changes; pair with the dashboard view of current RBAC. Verify trust-page certification claims are still current [REQUIRES-VERIFICATION]. Surface evidence in a one-pager for the auditor. | Pass / FailAi Platformhigh |
| 03 | An admin disables the PII guardrail at 3am; security wants to know who and when. | Per docs, admin actions on configs / virtual keys / guards / RBAC are captured in the audit log with actor, action, timestamp, and target. Stream the audit log to a SIEM and alert on guardrail-state changes outside business hours. | Pass / FailAi Platformcritical |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Portkey
- Ai Platform
- Safety Rbac And Governance
Recommended for
Works with
Related evals
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.