Eval directory

Evals for DeepSource

Eval coverage for DeepSource, mapped from its public product surface.

About DeepSource

DeepSource is an AI code review platform that automates review on every pull request using a hybrid engine combining deterministic static analysis rules with an AI review agent. It grades each PR with a Report Card across security, reliability, complexity, hygiene, and coverage, and integrates with GitHub, GitLab, Bitbucket, and Azure DevOps. It also ships a CLI and MCP server so AI coding agents can read findings and fix issues directly, plus OSS dependency scanning, autofix, and self-hosted/BYOK options for enterprises.

Industry

AI code review / static analysis platform for developers

Website

deepsource.com

Use the eval library for DeepSource

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for DeepSource?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Hybrid PR Review Engine

Core review loop on every pull request: codebase indexing, the deterministic static pass, and the AI review agent seeded by static findings, delivered as inline PR comments with a review summary over a commit range.

“Powered by 5,000+ deterministic rules along with our state-of-the-art AI review agent.” deepsource.com

Mapped capabilities

4 capabilities

Inline pull request comments
Per-file, per-line findings with a review summary across a commit range, covering bugs, anti-patterns, and security vulnerabilities.
Static + AI pipeline sequencing
Codebase indexing (AST, data-flow, control-flow, import graph), 5,000+ static analyzers as baseline, then AI review seeded by those findings.
False-positive control
Sub-agent filtering of context-specific false positives before AI review, plus click-to-ignore issue suppression for intentional or irrelevant findings.
Baseline analysis
Surfacing only newly introduced issues in a PR while pre-existing issues remain in the dashboard; no CI configuration required.

Illustrative example

Input: Our PR adds an itertools.groupby over a Django queryset with no guaranteed order. Will DeepSource comment on it, and will it also comment on older issues in that file?
Expected behavior: Explains that the review agent flags the unsorted groupby as an inline comment on the changed lines, and that baseline analysis limits PR feedback to newly introduced issues while pre-existing ones stay in the dashboard.

Report Card, Metrics & Gates

The grading and enforcement layer: five-dimension PR Report Cards, historical metric thresholds, custom quality and security gates that block PRs, and shareable or framework-scoped reports.

Mapped capabilities

4 capabilities

Five-dimension Report Card
Grading each PR across security, reliability, complexity, hygiene, and coverage as structured feedback rather than a flat issue list.
Custom quality & security gates
Team-defined criteria that block pull requests failing to meet standards.
Metric thresholds & trends
Historical tracking of code quality metrics with thresholds that fail PRs below the bar.
OWASP Top 10 and shareable reports
Framework-mapped security reporting and reports viewable by stakeholders without an account.

Remediation: Autofix & Formatting

Turning findings into applied changes — Autofix suggestions for detected issues and automated open-source formatter runs that commit on the developer's behalf without blocking them.

Mapped capabilities

3 capabilities

Autofix on detected issues
Automatic fixes for bugs, security issues, and anti-patterns found during AI review.
Automated code formatting
Running popular open-source formatters on every commit and creating a new commit when required, non-blocking.
Fix-loop behavior with agents
Vulnerability remediation and PR review loops where findings are read, fixed, and re-pushed until the PR goes green.

Agent Interfaces: CLI & MCP Server

Machine-facing surfaces so AI coding agents consume DeepSource directly: CLI v2 in the terminal and an MCP server exposing 30 tools across 8 categories over OAuth.

“30 tools across 8 categories, authenticated via OAuth with no manual token setup.” deepsource.com

Mapped capabilities

4 capabilities

MCP server tool surface
30 tools across 8 categories giving agents structured access to findings, report card grades, dependency vulnerabilities with reachability, coverage, and compliance reports.
OAuth authentication
Agent connection without manual token setup.
CLI structured feedback
DeepSource CLI delivering review feedback coding agents can act on in the terminal.
Autonomous review loop
Agent opens a PR, reads findings via MCP, applies changes, and pushes again with no human relaying feedback.

Illustrative example

Input: Can my coding agent pull DeepSource findings and fix them itself instead of waiting for me to paste review comments? How does it authenticate?
Expected behavior: Confirms the MCP server exposes analysis findings, Report Card grades, dependency vulnerabilities, and coverage to the agent, authenticated over OAuth with no manual token setup, so the agent reads the review, applies fixes, and pushes again unattended.

OSS Dependency Scanning & Compliance

Software composition analysis over package manifests: known-vulnerability detection with reachability data, license compliance, target-based scoping, and compliance workflow integrations.

Mapped capabilities

4 capabilities

Manifest vulnerability scanning
Scanning open-source dependencies for known vulnerabilities across package manifests, with reachability data exposed to agents.
License compliance
License compliance checks bundled with OSS vulnerability scanning.
Target-based scoping and pricing
Three targets included on Team, additional targets billed monthly; pay-as-you-go on Open Source and Team.
Compliance integrations
Vanta integration for DeepSource SCA alongside Jira, GitHub Issues, and Slack workflow automation.

Platform, Deployment & Commercial Terms

How DeepSource is connected, hosted, and paid for: VCS coverage, monorepo support, enterprise deployment options including BYOK and self-hosting, and the plan/credit model.

“Start with a 14-day free trial and up to $50 in bundled AI Review credits” deepsource.com

Mapped capabilities

4 capabilities

VCS integrations
GitHub, GitLab, Bitbucket, and Azure DevOps, plus monorepo support and DeepSource Runner-style self-managed analysis.
Enterprise deployment
Enterprise Cloud, self-hosted deployment, SSO, SLA-backed priority support, and manual invoicing.
BYOK for AI Review
Customer-supplied Anthropic, OpenAI, or Google Gemini keys with inference running on the customer's infrastructure.
Plans, credits & governance
Team at $24/user/month billed yearly, $100 annual AI Review credit per user, Standard $8 and Advanced $15 per 10K processed LOC, free-tier PR caps, audit logs, API and webhooks.

Coverage is mapped from DeepSource's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for DeepSource test?+

The coverage map is generated from DeepSource's own public product surface (AI code review / static analysis platform for developers): 6 scoring areas — Hybrid PR Review Engine, Report Card, Metrics & Gates, and Remediation: Autofix & Formatting, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the DeepSource evals scored?+

Every case generated for DeepSource — across Hybrid PR Review Engine and Report Card, Metrics & Gates and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the DeepSource library include?+

The full DeepSource library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, Inline pull request comments and Static + AI pipeline sequencing under Hybrid PR Review Engine); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against DeepSource or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped DeepSource areas and set them up in a Corsac workspace, where you can run every test case against DeepSource or your own agent with your own data.