Eval directory

Evals for Postman

Eval coverage for Postman, mapped from its public product surface.

About Postman

Postman is a unified platform for designing, testing, distributing, documenting, and monitoring APIs. It combines an API client, Spec Hub design and governance, Flows visual workflows, workspaces for internal/partner/public collaboration, monitors, and webhook tooling. An AI layer ("Postman AI" / "AI Engineer") generates specs, tests, docs, and Flows, and triages failed monitor runs, with paid tiers metered in AI credits.

Industry

API development and testing platform

Website

www.postman.com

Use the eval library for Postman

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Postman?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

API Client and Request Execution

The core client surface: composing and sending requests across protocols, handling auth and complex bodies, and inspecting responses.

Mapped capabilities

4 capabilities

Multi-protocol request support
Sending and configuring requests beyond REST as advertised by the multi-protocol client.
Built-in authentication and certificates
Selecting and configuring auth schemes, cookies, and client certificates for a request.
Variables and environments
Resolving variables across scopes and switching environments for the same request.
Response inspection and visualization
Reading response bodies, headers, and visualizations, plus request history.

API Design and Governance (Spec Hub)

Designing specifications in multiple formats and enforcing organizational standards on them.

Mapped capabilities

4 capabilities

Multi-format spec authoring
Designing REST, event-driven, GraphQL, gRPC, and Smithy specs in the visual designer or code view.
Definition import and multi-file specs
Importing existing definitions and working with multi-file specifications.
Governance rules and validation
Configuring rules, surfacing compliance state, and enforcing standards via Postman CLI.
Documentation preview and generation
Previewing schema-driven docs and generating endpoint documentation.

Testing, Monitoring, and Failure Triage

Running the tests a team already has — locally, in CI, on a schedule, and inside private networks — and reacting when they fail.

“Postman is a unified platform for designing, testing, distributing, documenting, and monitoring APIs.” www.postman.com

Mapped capabilities

4 capabilities

Collection Runner and CI execution
Running collections manually, via Postman CLI, and through CI integrations.
Scheduled and multi-region monitors
Configuring monitors, regions, static IP allowlisting, and reading pass/fail and latency results.
Private and VPC monitoring
Deploying an agent to monitor internal APIs that are not publicly reachable.
AI triage of failed runs
Reviewing a failed monitor run for root cause and a recommended fix.

Illustrative example

Input: Our checkout API only resolves inside our VPC. Can Postman monitors run against it, and what do we deploy?
Expected behavior: Confirms this is supported and says a Postman agent is deployed inside the VPC or private network so monitors can reach internal APIs, rather than routing from Postman's public regions.

Flows and Webhook Workflows

Visual, executable API workflows and first-class webhook capture, forwarding, and replay.

Mapped capabilities

4 capabilities

Flow authoring and execution
Building a canvas from existing collections and running it against an environment.
AI-generated Flows
Producing a working Flow from a prompt, including data mapping between steps and error handling.
Webhook capture and inspection
Provisioning a listener URL and inspecting incoming payloads and provider handshakes.
Forwarding and replay
Forwarding events to localhost or a chosen URL and replaying a captured event.

Workspaces, Collaboration, and Distribution

How API work is shared internally, with partners, and publicly, and how access is controlled.

“Deploy a Postman agent inside your VPC or private network to monitor internal APIs” www.postman.com

Mapped capabilities

4 capabilities

Workspace types and sharing
Choosing internal, partner, multi-partner, or public workspaces and sharing them appropriately.
Roles, permissions, and RBAC tiers
Assigning roles and understanding basic versus advanced access controls.
Version control and change visibility
Native Git, versioning, comments, changelog, and activity feed.
Public and Private API Network publishing
Publishing versions, universal search, team verification, and SDK distribution.

Postman AI and Plan Entitlements

The AI layer that generates specs, tests, docs, and Flows, and the plan boundaries that meter it.

Mapped capabilities

4 capabilities

AI credit allocation by plan
Monthly credits per tier, including pooled Enterprise credits.
AI-generated tests and documentation
Producing tests and docs grounded in existing collections and specs.
Feature gating across tiers
Which capabilities require Solo, Team, or Enterprise, including add-ons.
Enterprise controls and reporting
API Catalog, audit logs, governance reporting, Insights, and private runners.

Illustrative example

Input: How many Postman AI credits per month does each plan include, and which tier's credits are pooled across the organization?
Expected behavior: States 50 credits on Free, 400 per month on Solo, 400 per user on Team, and 800 per user on Enterprise, and identifies the Enterprise allocation as the pooled one.

Coverage is mapped from Postman's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Postman test?+

The coverage map is generated from Postman's own public product surface (API development and testing platform): 6 scoring areas — API Client and Request Execution, API Design and Governance (Spec Hub), and Testing, Monitoring, and Failure Triage, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Postman evals scored?+

Every case generated for Postman — across API Client and Request Execution and API Design and Governance (Spec Hub) and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Postman library include?+

The full Postman library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Multi-protocol request support and Built-in authentication and certificates under API Client and Request Execution); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Postman or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Postman areas and set them up in a Corsac workspace, where you can run every test case against Postman or your own agent with your own data.