Eval directory

Evals for Keycard

Eval coverage for Keycard, mapped from its public product surface.

About Keycard

Keycard is a control plane that governs what autonomous AI agents can access across tools, APIs, and data, issuing scoped, ephemeral credentials bound to the user, agent, and task. It provides identity, policy (RBAC/ABAC/ReBAC, app binding, zone and org policies), and real-time telemetry/audit of every agent action. TypeScript and Python SDKs support LangChain, MCP, A2A, or any API, with usage-based pricing metered in transactions.

Industry

AI agent identity and access control plane

Website

www.keycard.ai

Use the eval library for Keycard

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Keycard?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Agent Identity & Credential Lifecycle

Establishing identity for users, devices, agents, and workloads from the runtime environment, and issuing scoped, ephemeral credentials bound to user, agent, and task rather than static API keys.

“issuing ephemeral, identity-bound credentials at every hop” www.keycard.ai

Mapped capabilities

4 capabilities

Runtime-derived agent and workload identity
Agents establish identity from their deployment environment (Vercel, Cloudflare, Fly.io, AWS, GCP, Azure) with no secrets on disk.
Scoped, short-lived credential issuance
Credentials bound to the user, agent, and task, with narrow scope and an expiry such as 15m with optional auto-refresh.
Credential rotation and revocation
SDK-managed issuance, storage, rotation, and revocation so callers do not manage token lifecycles.
Identity provider types
Application, user, and workload identity across plans; device identity at the enterprise tier, including SSO and mTLS device binding.

Illustrative example

Input: Under a policy allowing logs:read during bug-fix tasks, an agent requests Datadog log access for task KYC-847 on behalf of a user authenticated via SSO.
Expected behavior: Keycard issues a credential limited to datadog/logs read-only, bound to the requesting user, the agent, and the task, and carrying a short expiry rather than an open-ended lifetime.

Policy Authoring & Access Decisions

Expressing and evaluating authorization rules across RBAC, ABAC, and ReBAC, plus Keycard's app binding, zone, org, and device-based policy layers that decide allow, deny, or step-up on each request.

Mapped capabilities

4 capabilities

RBAC, ABAC, and ReBAC evaluation
Authorization models available on every plan and applied to a concrete resource and action.
App Binding and Dependencies policy
Constraining which app a credential belongs to and which downstream dependencies it may reach.
Task-conditioned allow and forbid rules
Rules keyed to task context and resource attributes, e.g. forbid when resource.tags.sensitivity == "pii" while allowing logs:read during a bug fix.
Zone, Org, and Device-Based Policy tiers
Higher-scope policy layers gated by plan: Zone Policy on Team, Org and Device-Based Policy on Enterprise.

Illustrative example

Input: An agent working the task "fix rate limit bug" requests read access to prod-db customers, a resource tagged sensitivity = pii, under a policy that forbids PII without elevated approval.
Expected behavior: The request is denied rather than granted or silently narrowed. The response names the blocked resource and the reason — PII requires elevated approval — and does not issue a credential for any part of that resource.

Multi-Agent Delegation & Handoffs

Keycard for Multi-Agent Apps: preserving scoped, identity-bound access across fan-out, peer-to-peer, and chained agent collaboration spanning clouds and trust boundaries.

“Control agent access to tools, APIs, and data” www.keycard.ai

Mapped capabilities

4 capabilities

Per-hop credential issuance
A fresh, task-scoped credential at every collaboration or tool-calling hop rather than one broad credential for the chain.
On-behalf-of context propagation
Carrying the originating human and delegating agent through the chain, e.g. Sarah → Claude → KYC-847.
Cross-boundary agent-to-agent access
A2A and cross-cloud handoffs where each agent's access stays scoped to the task at hand.
Token exchange and cross-app access standards
Standards-based exchange including ID-JAG and cross-app access from Okta.

Telemetry, Audit & Investigation

Real-time visibility into every agent action, who authorized it, and what the policy decided — including denied attempts — with retention and export paths for security tooling.

“See every action your agents take. Know exactly who authorized it.” www.keycard.ai

Mapped capabilities

4 capabilities

Per-action decision records
Each request logged with identity, agent, task, policy matched, granted scope, and outcome.
Denied and flagged event handling
Blocked requests recorded, flagged to the security team, and streamed to SIEM.
Retrospective investigation queries
Answering natural-language questions such as "Who accessed prod last night" from telemetry, including via Claude Code.
Retention and continuous export
7-day retention on Starter, configurable retention on Team, continuous OCSF export on Enterprise.

SDK & Framework Integration

Adding scoped auth to existing agent code through TypeScript and Python SDKs with out-of-the-box support for common agent frameworks and protocols, without requiring identity expertise.

Mapped capabilities

4 capabilities

TypeScript and Python SDK surface
Language SDKs that wrap credential lifecycle and access requests for application code.
LangChain, MCP, and A2A support
Out-of-the-box integration paths for the named frameworks and protocols.
Arbitrary API and tool protection
Applying Keycard-mediated access to any API the agent calls, not just pre-integrated tools.
Deployment targets without stored secrets
Running on Vercel, Cloudflare, Fly.io, AWS, GCP, or Azure with no credentials in databases or on disk.

Plans, Metering & Enterprise Deployment

The commercial and operational envelope: transaction-based metering, plan-gated capabilities, identity provisioning, and deployment and key-management options for regulated organizations.

Mapped capabilities

4 capabilities

Transaction metering semantics
A transaction recorded per credential issuance, access validation, credential exchange, or step-up approval.
Plan allocations and overage
Starter's 5,000/mo hard cap, Team's $500/mo with 100,000 included, and $1 per 1,000 additional transactions on paid plans.
Organization provisioning and SSO
SSO for org users on Team; SCIM, Active Directory, and LDAP provisioning on Enterprise.
Deployment and key custody options
Dedicated, BYOC, or on-prem deployment with private networking and customer-managed KMS.

Coverage is mapped from Keycard's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Keycard test?+

The coverage map is generated from Keycard's own public product surface (AI agent identity and access control plane): 6 scoring areas — Agent Identity & Credential Lifecycle, Policy Authoring & Access Decisions, and Multi-Agent Delegation & Handoffs, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Keycard evals scored?+

Every case generated for Keycard — across Agent Identity & Credential Lifecycle and Policy Authoring & Access Decisions and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Keycard library include?+

The full Keycard library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Runtime-derived agent and workload identity and Scoped, short-lived credential issuance under Agent Identity & Credential Lifecycle); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Keycard or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Keycard areas and set them up in a Corsac workspace, where you can run every test case against Keycard or your own agent with your own data.