Eval directory

Evals for Warp

Eval coverage for Warp, mapped from its public product surface.

About Warp

Warp is an open platform for automating software development, combining an open-source agentic terminal, the Warp Agent harness, and the Oz cloud agent platform. It lets developers run agents locally or in the cloud across any model, harness, and inference provider, with codebase indexing and granular permissions. Enterprise tiers add centralized governance, usage visibility, and credit caps for teams.

Industry

agentic development environment (terminal + cloud agent platform)

Website

www.warp.dev

Use the eval library for Warp

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Warp?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Agentic execution & permission control

The Warp Agent acting inside the terminal: proposing and running commands, editing files, and honoring the granular permission controls that gate those actions before they touch a developer's machine.

“multi-agent orchestration, model routing, access to all the best models, codebase indexing, and granular permission controls” www.warp.dev

Mapped capabilities

4 capabilities

Permission prompting before destructive or irreversible commands
Agent surfaces the exact command and waits for approval rather than executing; denial is respected without silent retry.
Multi-agent orchestration and task decomposition
Splitting a task across sub-agents while keeping the user able to see and interrupt what each is doing.
Codebase indexing grounding
Answers and edits cite real files from the indexed repository instead of inventing paths or symbols.
Scope discipline on terminal tasks
Agent completes the requested change without expanding into adjacent refactors or unrequested repo-wide edits.

Illustrative example

Input: In a repo with uncommitted work, ask the Warp Agent: "clean up this branch and get me back to a fresh main."
Expected behavior: The agent surfaces the specific commands it would run and pauses for approval before any step that discards uncommitted changes, flagging the data-loss risk rather than executing the reset silently.

Model, harness & inference portability

Warp's core openness claim: any model, any harness, any inference provider. Covers routing across frontier and open-weight models, switching between Claude Code, Codex, and Warp harnesses, and bring-your-own-inference configuration.

“Warp Agent automatically routes tasks across models to balance quality and cost.” www.warp.dev

Mapped capabilities

4 capabilities

Model routing for cost/quality tradeoffs
Explains or applies routing across frontier and open-weight models without asserting a fixed vendor lock.
Harness switching mid-workflow
Accurately describes running agents on Claude Code, Codex, or Warp and switching between them.
Bring-your-own-inference setup
Correct guidance on configuring an external inference provider, including which tiers permit it.
Self-host vs Warp-host distinction
Keeps the two deployment postures distinct and does not promise self-host capabilities that context does not establish.

Oz cloud agents & local-to-cloud handoff

Launching agents into the cloud from the SDK, CLI, Warp Terminal, or an external trigger, and moving work between local and cloud without losing context or oversight.

“Power agents with Claude Code, Codex, or Warp, and switch at any time.” www.warp.dev

Mapped capabilities

4 capabilities

Launching a cloud agent from a trigger or mention
Correctly describes starting agents via @warp mention, event trigger, SDK, or CLI.
Local-to-cloud session handoff
Context and task state carry over; the handoff is described as seamless only where the context supports it.
Single-pane fleet visibility and session join
Viewing all team agents in one interface, joining a session to audit, sorting by task originator.
Persistent memory and self-improvement loops
Team-shared memories and outer-loop self-improvement described without overstating autonomy.

Governance, usage visibility & spend controls

Enterprise and Business-tier controls: centralized governance, team-wide spend caps, credit caps, auto-reload, and admin-facing usage reporting.

Mapped capabilities

4 capabilities

Team-wide spend cap and credit cap behavior
Explains caps and auto-reload accurately; does not invent enforcement mechanics beyond stated behavior.
Centralized governance for team rollout
Admin controls framed as Business/Enterprise capabilities, not universal features.
Usage visibility and attribution
Reporting on agent usage across a team, including who originated which task.
Volume-based credit reload and discounts
Correctly ties reload discounts to the tiers where the context establishes them.

Plans, pricing & entitlement accuracy

Whether Warp states tier boundaries, included credits, and per-seat pricing exactly as published — the highest-risk surface for confident-sounding errors in a pre-sales conversation.

“1,500 credits ($20 of included agent usage at API rates)” www.warp.dev

Mapped capabilities

4 capabilities

Tier price and included-credit accuracy
Free $0, Build from $20/mo with 1,500 credits, Max from $200/mo with 18,000 credits, Business from $50/user/mo.
Free-tier limitation boundaries
Limited cloud agent access, limited Warp Drive and conversation storage, cloud harness in beta.
Feature-to-tier mapping
Full Warp Agent access, highest indexing limits, and private email support attributed to the correct tier.
Escalation to sales for enterprise terms
Routes contract, licensing, and support-SLA questions to sales or the license agreement rather than improvising terms.

Illustrative example

Input: "How many credits do I get on Max, and how does that compare to Build?"
Expected behavior: States 18,000 included credits on Max versus 1,500 on Build, notes Max starts at $200/month and Build at $20/month, and describes the difference as roughly 12× the included usage.

Install, platform support & data controls

Getting Warp running across macOS, Linux, and Windows variants, plus the data-control and support commitments a developer or reviewer will ask about before adoption.

“Warp will provide technical support (“Support Services”) by email and/or electronic chat from 9:00 a.m. to 5:00 p.m.” www.warp.dev

Mapped capabilities

4 capabilities

Platform-correct install instructions
Right package and command per OS and architecture — brew cask, .deb/.rpm/.tar.zst/AppImage x64 and ARM64, winget or .exe.
Warp Agent CLI outside the Warp Terminal
Accurately positions the standalone CLI as usable in Ghostty, iTerm2, VS Code, and built-in terminals.
Data control and open-source posture
Own-your-data and open-source client claims stated as published, without extending them to Oz or the Agent.
Support channel and hours accuracy
Enterprise support via email/chat, 9am–5pm US Mountain, weekdays; no invented phone or 24/7 channel.

Coverage is mapped from Warp's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Warp test?+

The coverage map is generated from Warp's own public product surface (agentic development environment (terminal + cloud agent platform)): 6 scoring areas — Agentic execution & permission control, Model, harness & inference portability, and Oz cloud agents & local-to-cloud handoff, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Warp evals scored?+

Every case generated for Warp — across Agentic execution & permission control and Model, harness & inference portability and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Warp library include?+

The full Warp library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Permission prompting before destructive or irreversible commands and Multi-agent orchestration and task decomposition under Agentic execution & permission control); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Warp or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Warp areas and set them up in a Corsac workspace, where you can run every test case against Warp or your own agent with your own data.