Eval directory

Evals for Sourcery

Eval coverage for Sourcery, mapped from its public product surface.

About Sourcery

Sourcery is an automated code review tool that reviews pull requests, enforces code standards, and scans repositories for security vulnerabilities across GitHub, GitLab, and IDEs. It provides PR summaries, line-by-line reviews, custom review rules, repo analytics, and fix instructions that can be handed to an AI agent. Plans range from free open-source use through per-seat Pro and Team tiers to an Enterprise tier with a self-hosting option.

Industry

AI code review and security scanning for developer teams

Headquarters

United Kingdom

Website

www.sourcery.ai

Use the eval library for Sourcery

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Sourcery?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Pull Request Review

Behavior of the review bot on an individual pull request: change summaries, diagrams, line-by-line comments, and the logic and edge-case issues it surfaces before merge.

“Security checks from the first line of code to the final merge.” www.sourcery.ai

Mapped capabilities

4 capabilities

PR summaries
Generates a summary of what a change does, scoped to the diff rather than the whole repository.
Change diagrams
Produces Mermaid-style diagrams of code changes where the change structure supports one.
Line-by-line comments
Anchors review comments to the correct file and line within the diff.
Bug and edge-case detection
Flags logic errors and unhandled edge cases introduced by the change.

Repository Security Scanning

Continuous scanning across repositories and on new PRs, covering the scan types listed in the Dec 2025 changelog and the triage workflow described on the security page.

“Scan your code daily for hundreds of security vulnerabilities” www.sourcery.ai

Mapped capabilities

4 capabilities

Scan type coverage
SAST, SCA, secrets, IaC, and license detection as distinct finding categories.
Issue grouping and triage
Groups related findings and points to a root cause to support risk assessment.
PR security gating
Flags security risk on new PRs and blocks merges when configured to do so.
Noise controls
Honors configuration that ignores specific paths and rules.

Illustrative example

Input: A pull request adds a Python module containing a hardcoded API key assigned to a module-level constant and passed to an outbound HTTP client, on a repo configured to block PRs with vulnerabilities.
Expected behavior: Sourcery flags the hardcoded credential as a secrets finding on the line that introduces it, explains the exposure risk, and reports a blocking status on the PR rather than a non-blocking comment.

Code Standards and Custom Rules

Enforcement of team-specific conventions through custom review rules and uploaded style guides, and consistency of that enforcement across reviews.

“Automated code reviews designed for security and speed.” www.sourcery.ai

Mapped capabilities

4 capabilities

Custom review rules
Applies user-defined rules to matching code in a PR.
Style guide adherence
Reviews against a supplied style guide rather than generic defaults.
Rule scoping
Applies rules only to the languages, paths, or repos they are configured for.
Enforcement consistency
Reaches the same verdict on the same violation across repeated reviews.

Fix Guidance and Agent Handoff

The remediation half of the product: explaining each finding and emitting fix instructions that can be handed to an external AI coding agent.

Mapped capabilities

3 capabilities

Vulnerability explanations
Explains why a flagged issue is a risk, tied to the specific code.
Agent-ready fix instructions
Produces instructions concrete enough to hand to an AI agent of the user's choice.
Fix scope discipline
Keeps proposed fixes limited to the flagged issue.

Platform and IDE Integration

Delivery surfaces documented across the site and changelog: GitHub and GitLab including self-hosted instances, VS Code and JetBrains plugins, and CLI usage.

Mapped capabilities

4 capabilities

GitHub and GitLab reviews
Operates as a review integration on both hosts, including self-hosted instances.
IDE review and chat
Delivers reviews and chat responses inside VS Code and JetBrains.
One-click apply
Applies a suggested change from the IDE chat surface.
Surface consistency
Reports the same issue consistently whether reviewed in the IDE or on the PR.

Plans, Entitlements, and Code Handling

Seat-based tiers and their stated limits, plus the code-privacy and LLM-provider commitments in the pricing FAQ and privacy policy.

“Sourcery is fully free to use on open source projects.” www.sourcery.ai

Mapped capabilities

4 capabilities

Tier limit enforcement
Repo caps, scan cadence, and review rate limits differ correctly across Open Source, Pro, and Team.
Seat assignment
Charges only for developers assigned seats, per the pricing FAQ.
Open source access
Works on public repos without a paid plan.
Code handling claims
Answers about code storage, LLM providers, and self-hosting match the published policy.

Illustrative example

Input: An account on the Pro plan attempts to enable security scanning on an eleventh private repository.
Expected behavior: Sourcery declines to enable the eleventh repo, states that Pro covers limited security scans for ten repos, and points to the Team tier for broader coverage instead of silently enabling it.

Coverage is mapped from Sourcery's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Sourcery test?+

The coverage map is generated from Sourcery's own public product surface (AI code review and security scanning for developer teams): 6 scoring areas — Pull Request Review, Repository Security Scanning, and Code Standards and Custom Rules, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Sourcery evals scored?+

Every case generated for Sourcery — across Pull Request Review and Repository Security Scanning and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Sourcery library include?+

The full Sourcery library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, PR summaries and Change diagrams under Pull Request Review); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Sourcery or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Sourcery areas and set them up in a Corsac workspace, where you can run every test case against Sourcery or your own agent with your own data.