Eval directory

Evals for cubic

Eval coverage for cubic, mapped from its public product surface.

About cubic

cubic is an AI code reviewer that analyzes pull requests and entire codebases to surface bugs, tech debt, and coding-rule violations. It works inside GitHub and the IDE, adding inline review comments, AI-generated PR descriptions, one-click fixes, and an AI wiki over the codebase. Plans range from a free tier with limited monthly PR reviews up to Team, Pro, and Enterprise tiers with codebase scans, integrations, and analytics.

Industry

AI code review tool for engineering teams

Website

www.cubic.dev

Use the eval library for cubic

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for cubic?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Pull Request Review Accuracy

Core inline review on a diff: finding real defects while keeping false positives low, which is the precision/recall tradeoff cubic markets via its F1 benchmark positioning.

“cubic finds hard-to-find bugs in pull requests and your entire codebase.” www.cubic.dev

Mapped capabilities

4 capabilities

Real-defect detection in a changed diff
Flags genuine bugs introduced by the PR, anchored to the correct file and line.
False-positive restraint
Withholds comments on correct code and on nitpicks that do not affect behavior.
Cross-file and repo-context reasoning
Uses surrounding codebase context, not just the diff hunk, when a change breaks a distant caller.
Severity and comment prioritization
Distinguishes bugs from tech debt so high-impact findings are not buried.

Illustrative example

Input: A pull request diff adds a function that reads a field from an optional lookup result before checking whether the lookup returned anything, plus unrelated formatting changes in another file.
Expected behavior: cubic posts one inline comment on the unguarded dereference, anchored to that line, explaining the null case. It does not comment on the formatting-only changes.

Review Rules & Custom Agents

Team-defined review policy: plain-English rules and a bounded number of custom agents that enforce conventions consistently across every PR.

“Catch bugs, tech debt, and enforce your coding rules.” www.cubic.dev

Mapped capabilities

4 capabilities

Plain-English rule enforcement
Applies a natural-language rule to the diff and comments only when the rule is actually violated.
Custom agent configuration and scoping
Runs the configured agents against the PR within the plan's agent count limit.
Rule conflict and precedence handling
Behaves predictably when a custom rule overlaps or conflicts with default review behavior.
Consistency across repeat violations
Enforces the same rule the same way on later PRs rather than flagging it once.

Illustrative example

Input: A repo rule reads: 'No raw SQL string concatenation; use the query builder.' The PR adds one concatenated SQL string and one correct query-builder call.
Expected behavior: cubic flags the concatenated SQL string and cites the configured rule. It leaves the compliant query-builder call uncommented.

Fix Generation & Remediation

Turning a finding into applied code: one-click commits for simple fixes, 'Fix with cubic' for harder ones, and auto-created fix PRs from scans on higher tiers.

“cubic reviews code in GitHub and your IDE.” www.cubic.dev

Mapped capabilities

4 capabilities

One-click fix correctness
The committed suggestion compiles conceptually and resolves the flagged issue without collateral edits.
Handoff to coding agents for complex fixes
Escalates non-trivial findings to the agent path instead of emitting a wrong inline patch.
Auto-created fix pull requests
Generates a scoped fix PR from a scan finding with an accurate description of what changed.
Declining to fix when unsafe
Leaves ambiguous or design-level findings to the developer rather than force-applying a change.

Codebase Scans & AI Wiki

Whole-repository surfaces beyond the PR: scheduled codebase scans, the generated AI wiki over the repo, and their MCP access points.

Mapped capabilities

4 capabilities

Scheduled scan findings on untouched code
Surfaces latent bugs and tech debt outside any active pull request.
Wiki grounding in actual repository contents
Wiki pages describe code that exists rather than plausible-sounding architecture.
Wiki refresh after code changes
Updated pages reflect merged changes on the plan's daily or weekly cadence.
Scan and wiki MCP access
Exposes scan results and wiki content to external tools through the documented MCP surfaces.

Context Integrations & PR Comprehension

Pulling intent from outside the repo — Jira, Linear, Asana, Notion, Confluence, and custom context — and producing AI PR descriptions that explain impact.

“AI PR descriptions that understand changes and highlight impact.” www.cubic.dev

Mapped capabilities

4 capabilities

AI PR description accuracy
Summarizes what the diff actually changes and highlights user-visible impact.
Ticket-to-diff intent checking
Compares linked issue-tracker intent against the code and notes divergence.
Custom context ingestion
Applies team-supplied context documents to the review instead of ignoring them.
Missing or stale integration data
Degrades gracefully when a linked ticket or doc is unavailable.

Plan Entitlements & Enterprise Controls

Tier-bound limits and admin surfaces published on the pricing page: monthly PR review caps, lines-of-code allowances, agent counts, analytics, and enterprise access controls.

“40k lines of code reviewed per developer” www.cubic.dev

Mapped capabilities

4 capabilities

Free-tier PR review quota enforcement
Honors the 20 free monthly reviews and communicates the limit clearly at exhaustion.
Lines-of-code allowance per developer
Applies the Team and Pro per-developer review volume limits.
Feature gating by tier
Restricts CLI, scans, Confluence, and analytics features to the plans that include them.
Enterprise access controls
Supports SSO/SAML-based administration on the enterprise tier.

Coverage is mapped from cubic's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for cubic test?+

The coverage map is generated from cubic's own public product surface (AI code review tool for engineering teams): 6 scoring areas — Pull Request Review Accuracy, Review Rules & Custom Agents, and Fix Generation & Remediation, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the cubic evals scored?+

Every case generated for cubic — across Pull Request Review Accuracy and Review Rules & Custom Agents and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the cubic library include?+

The full cubic library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Real-defect detection in a changed diff and False-positive restraint under Pull Request Review Accuracy); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against cubic or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped cubic areas and set them up in a Corsac workspace, where you can run every test case against cubic or your own agent with your own data.