Eval directory

Evals for Greptile

Eval coverage for Greptile, mapped from its public product surface.

About Greptile

Greptile is an AI code review agent that builds a graph index of a repository and uses parallel agents to review every pull request with full codebase context. It posts findings as PR comments with suggested fixes, learns team standards from engineers' reactions and replies, and can hand issues off to coding agents like Claude Code, Cursor, or Devin. Its TREX layer adds an execution step that runs code in a sandbox and attaches evidence such as logs, screenshots, and API traces.

Industry

AI code review agent for pull requests

Website

www.greptile.com

Use the eval library for Greptile

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Greptile?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Codebase Indexing & Cross-File Reasoning

Building and using the graph of files, functions, and dependencies so review reasons about the whole system rather than a file in isolation.

“AI agents that review and test pull requests with full context of the codebase.” www.greptile.com

Mapped capabilities

4 capabilities

Repository graph construction
Indexing files, functions, classes, and dependencies after the GitHub/GitLab app connects a repo.
Impact beyond the diff
Tracing how a change affects callers, dependents, and other files not present in the diff.
Multi-repository and multi-language scope
Behavior across the unlimited-repository model, including repos of differing size and language.
Index freshness
Keeping the graph aligned with the branch and base as the repository changes over time.

PR Review & Finding Quality

What the swarm of parallel agents posts back on a pull request: the finding, its severity class, the suggested fix, and restraint about noise.

“Posts findings in ~3 minutes as PR comments with suggested fixes.” www.greptile.com

Mapped capabilities

4 capabilities

Bug classes caught
Style violations, security risks, and multi-file logical bugs as described on the product site.
Suggested fixes in comments
Comments carrying file paths, line numbers, and concrete suggested code.
Review summary
The per-PR summary that aggregates issues and exposes the Fix All action.
Noise and false-positive restraint
Not commenting on issues the team has signaled it does not care about.

Illustrative example

Input: A pull request deletes a helper method from one module. The method is still called from a second file that the diff does not touch. Review the pull request.
Expected behavior: Greptile reports the broken call site in the untouched file, naming its path and line, and offers a suggested fix. It does not treat the change as safe simply because the caller is outside the diff.

TREX Execution & Evidence

The execution layer that selects behavior worth running, runs it in a sandbox, and attaches evidence instead of asserting a result.

Mapped capabilities

4 capabilities

Selecting what to run
Picking the behavior in the change that warrants execution and spinning up a TREX agent.
Sandboxed execution
Running the relevant code in a sandbox using the shared review context, without extra scaffolding.
Artifact attachment
Attaching logs, screenshots, API traces, execution scripts, or a video of a UI change to the comment.
Evidence-backed claims
Reporting results so a human or downstream agent can confirm them rather than take them on faith.

Illustrative example

Input: A pull request changes an HTTP endpoint's response handling. TREX is enabled on the organization. Review and test the pull request.
Expected behavior: TREX runs the affected endpoint in a sandbox and attaches evidence for its conclusion, such as a log, an API trace, or a screenshot. It does not report a pass or fail without an artifact backing it.

Personalization & Team Standards

How Greptile adapts to a specific team through explicit rules and implicit signals from engineers.

“Greptile builds a complete graph of your codebase - every function, class, and dependency.” www.greptile.com

Mapped capabilities

4 capabilities

Custom rules
Team-authored review rules available on the Pro and Enterprise plans.
Learning from reactions and replies
Using thumbs-up/thumbs-down reactions and comment replies to shift what gets flagged.
Importing existing AI rules files
Picking up a repository's existing agent rules files during onboarding.
Standards inferred from the codebase
Reading other engineers' comments and repo conventions to infer coding standards over time.

Agent Handoff & Fix Workflow

Routing a finding out of the review and into a coding agent that can apply the change.

Mapped capabilities

4 capabilities

Fix with your Agent
Sending a single issue with file paths, line numbers, and suggested code to an external agent.
Supported agent targets
Claude Code, Codex, Conductor, Cursor, and Devin as documented handoff destinations.
Fix All from the summary
Dispatching every issue in a review at once from the summary comment.
Handoff payload fidelity
Preserving enough context in the handoff that the receiving agent can act without re-deriving it.

Onboarding, Deployment & Plan Limits

Getting connected and staying inside the documented deployment, access, and entitlement boundaries.

Mapped capabilities

4 capabilities

CLI onboarding
greptile onboard and greptile review, including the Node 22+ and CLI v3.2.0+ prerequisites and the old-build error path.
Repository connection
Installing the GitHub or GitLab app, creating the organization, and enabling repositories.
Self-hosting and enterprise controls
Docker/Kubernetes deployment, air-gapped environments, custom LLMs, SSO/SAML, and GitHub Enterprise.
Credits and plan entitlements
Standard reviews at 1 credit and TREX reviews at 3, seat and plan limits, and the free-for-OSS path.

Coverage is mapped from Greptile's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Greptile test?+

The coverage map is generated from Greptile's own public product surface (AI code review agent for pull requests): 6 scoring areas — Codebase Indexing & Cross-File Reasoning, PR Review & Finding Quality, and TREX Execution & Evidence, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Greptile evals scored?+

Every case generated for Greptile — across Codebase Indexing & Cross-File Reasoning and PR Review & Finding Quality and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Greptile library include?+

The full Greptile library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Repository graph construction and Impact beyond the diff under Codebase Indexing & Cross-File Reasoning); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Greptile or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Greptile areas and set them up in a Corsac workspace, where you can run every test case against Greptile or your own agent with your own data.