Eval directory

Evals for Graphite

Eval coverage for Graphite, mapped from its public product surface.

About Graphite

Graphite is an AI code review platform layered on GitHub that combines AI reviews, a chat assistant, stacked pull requests, a stack-aware merge queue, and a custom PR page and inbox. It ships a CLI, VS Code extension, MCP integration, Slack notifications, and developer productivity insights. Recent launches add Cursor Cloud Agents so users can create, review, and merge agent-generated PRs from within Graphite.

Industry

AI code review and pull-request workflow platform for engineering teams

Website

www.graphite.com

Use the eval library for Graphite

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Graphite?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

AI code review

Codebase-aware automated review on pull requests: what it flags, how it justifies findings, and the fixes it proposes pre-merge.

“Graphite is the AI code review platform where teams ship higher quality code, faster.” www.graphite.com

Mapped capabilities

4 capabilities

Bug and defect detection in a diff
Identifies correctness-affecting issues in changed code and points at the responsible lines.
Suggested fixes
Proposes concrete, applicable changes rather than only describing a problem.
Signal control and noise suppression
Prioritizes critical findings over low-value nits so review comments stay high signal.
Review customization
Honors team-configured review rules and customizations where the plan supports them.

Graphite Chat

The conversational assistant on the PR page that supplies context on changes, explains CI failures, and helps improve a PR in flow.

“Unlimited access to Graphite Chat Unlimited AI Reviews AI Review customizations Automations Merge Queue” www.graphite.com

Mapped capabilities

4 capabilities

Explaining a code change
Answers questions about what a diff does using repository context.
CI failure triage
Interprets a failing check and describes what broke and how to address it.
Code search grounding
Retrieves relevant code from the repo to support an answer instead of guessing.
PR improvement assistance
Turns review feedback into proposed edits on the open PR.

Stacked pull requests

Breaking large changes into small sequenced PRs and keeping the stack coherent through the CLI, VS Code extension, and MCP.

Mapped capabilities

4 capabilities

Creating and sequencing a stack
Splits work into dependent PRs in a defined order.
Restacking after an upstream edit
Propagates a change in a lower PR to the branches above it.
CLI command surface
Handles simplified git commands, including legacy pre-1.0 command names that now prompt for the new form.
Editor and MCP clients
Exposes the same stacking operations through the VS Code extension and MCP integration.

Merge queue and branch safety

Landing stacked work without conflicts: ordered merges, CI cost controls, and guardrails on shared branches.

“Our stack-aware merge queue lands PRs in order and keeps branches green” www.graphite.com

Mapped capabilities

4 capabilities

Stack-aware ordered landing
Merges PRs in dependency order and keeps branches green.
CI Optimizations skip decisions
Returns whether CI should be skipped for a PR given per-repo stack configuration.
Fail-safe defaults
Declines to skip CI on malformed or errored requests, when the feature is disabled, or when the PR is merging via the queue or as a stack.
Frozen branches
Restricts changes to branches teammates are building on.

Illustrative example

Input: A repo has CI Optimizations enabled. The CI step's request to the Graphite API is malformed and returns an error. Should this PR's CI run be skipped?
Expected behavior: The response states that CI is not skipped. On a malformed or errored request the documented behavior is to fail open and run CI, and the answer does not offer skipping as an option in this case.

Agent-generated PR workflow

Creating, reviewing, refining, and merging Cursor Cloud Agent work from the Agents tab and PR page without leaving Graphite.

“Get immediate, actionable feedback with Graphite's codebase-aware AI code review agent.” www.graphite.com

Mapped capabilities

4 capabilities

Kicking off an agent from a task description
Turns a described change into an agent run that opens a draft PR.
Reviewing agent output
Applies the same review and diff surfaces to agent-authored changes.
Iterating with an agent on the PR page
Requests follow-up changes to an open agent PR in place.
Model selection
Lets the user choose the model appropriate to the task.

Review surfaces, notifications, and plan boundaries

Where reviewers live day to day — the PR page, the multi-repo inbox, Slack, and insights — plus the entitlement and access limits that gate them.

Mapped capabilities

4 capabilities

PR inbox filters and rules
Surfaces the right PRs across many repositories using custom filters.
Slack notifications and in-Slack actions
Sends configured event notifications and supports approve, comment, request changes, or merge for PRs up to 25 lines.
Developer productivity insights
Reports team review and throughput metrics.
Plan entitlements and access control
Respects tier boundaries such as Merge Queue and unlimited AI reviews on Team, and ACLs, SAML, and audit log on Enterprise.

Illustrative example

Input: We're on the Starter plan at $20 per user per month. Can we turn on the Graphite merge queue for our org today, or do we need to change plans?
Expected behavior: The answer says Merge Queue is not included in Starter and requires upgrading to Team (or Enterprise for advanced merge queue settings). It does not claim the feature can be enabled on Starter.

Coverage is mapped from Graphite's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Graphite test?+

The coverage map is generated from Graphite's own public product surface (AI code review and pull-request workflow platform for engineering teams): 6 scoring areas — AI code review, Graphite Chat, and Stacked pull requests, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Graphite evals scored?+

Every case generated for Graphite — across AI code review and Graphite Chat and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Graphite library include?+

The full Graphite library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Bug and defect detection in a diff and Suggested fixes under AI code review); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Graphite or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Graphite areas and set them up in a Corsac workspace, where you can run every test case against Graphite or your own agent with your own data.