Eval directory

Evals for Korbit

Eval coverage for Korbit, mapped from its public product surface.

About Korbit

Korbit AI is an AI-powered code review tool that reviews pull requests in GitHub, GitLab, and Bitbucket, flagging bugs and explaining how to fix them. It also generates PR descriptions, offers chat with your codebase, and reports engineering metrics through Korbit Insights. Plans range from a per-user Pro tier to an Enterprise tier with on-prem installation, all starting with a 14-day free trial.

Industry

AI code review / developer tooling

Website

www.korbit.ai

Use the eval library for Korbit

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Korbit?

6 scoring areas · 22 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Code review issue detection

The core capability: analyzing a pull request diff with project context, surfacing real bugs and code issues, and explaining how to fix them.

“you retain full ownership of all code and suggestions provided during the review process” www.korbit.ai

Mapped capabilities

4 capabilities

Bug identification in a diff
Flags defects introduced by the changed lines rather than restating the diff.
Fix explanation quality
Each flagged issue comes with an actionable explanation of how to correct it.
Precision and hallucination control
Avoids inventing issues, symbols, or behavior not present in the reviewed code.
Low-value issue filtering
Suppresses noise and trivial nits so high-impact findings stay visible.

Illustrative example

Input: A pull request diff adds a Python function that calls `user.profile.email` immediately after a lookup that can return `None`, with no guard on the result.
Expected behavior: Korbit flags the unguarded dereference on the changed line as a potential runtime failure and explains a concrete fix, such as checking the lookup result before accessing the attribute.

Pull request workflow integration

How Korbit behaves inside GitHub, GitLab, and Bitbucket: when reviews trigger, what gets posted, and how it responds to developers in-thread.

“Works seamlessly in GitHub, GitLab, and Bitbucket” www.korbit.ai

Mapped capabilities

4 capabilities

Automatic vs. manual review triggering
Runs on the expected event and honors explicitly requested reviews.
PR description generation
Produces a clear summary of what the pull request changes.
In-thread comment replies
Responds to developer replies on its own review comments within the PR.
Cross-provider consistency
Comparable review behavior across GitHub, GitLab, and Bitbucket.

Review configuration and policy

Admin-facing controls that shape what Korbit reviews and how strictly: review modes, custom coding rules, repository scope, and feedback-driven tuning.

Mapped capabilities

4 capabilities

Comprehensive vs. Essential modes
Review depth and comment volume match the selected mode.
Custom policy enforcement
Applies a team's stated coding rules to the diff and cites the rule when flagging.
Repository access scoping
Reviews only repositories the customer has granted access to.
Feedback-driven adaptation
Incorporates prior developer feedback into later reviews of the same project.

Engineering metrics and reporting

Korbit Insights: reports on review activity and code quality for engineering leaders, including scheduled delivery into Slack on the Max plan.

Mapped capabilities

3 capabilities

Core Insights reports
Reports reflect the underlying review and PR activity for the selected scope.
Slack report delivery
Daily or weekly Insights summaries reach the configured Slack channel.
Console management views
Korbit Console surfaces users, repositories, and review activity for admins.

Codebase question answering

Ask Korbit: conversational questions about a customer's codebase, where grounding in actual repository content matters more than fluency.

Mapped capabilities

3 capabilities

Grounded answers about repository code
Answers cite or reflect code that exists in the connected repository.
Refusal on out-of-scope questions
Declines or asks for scope when the question is not answerable from the codebase.
Context carried across a conversation
Follow-up questions resolve against the earlier turns in the same thread.

Security, data handling, and plan boundaries

Claims a buyer will test: SOC 2 Type II, zero-day retention with model providers, code never used for training, code cleared after review, customer ownership of output, plus limits and entitlements across Pro, Max, and Enterprise.

“your code remains confidential and is never used for model training” www.korbit.ai

Mapped capabilities

4 capabilities

Code retention and training claims
States accurately that code is not retained for training and is cleared after review.
Ownership of code and suggestions
Confirms the customer retains ownership of code and of review suggestions.
Plan entitlement accuracy
Attributes custom policies, Slack Insights, and on-prem to the correct tier.
Large-PR and concurrency limits
Communicates limits rather than silently truncating a review.

Illustrative example

Input: A prospective Enterprise buyer asks: does Korbit use our proprietary code to train language models, and is our code stored after a review completes?
Expected behavior: States that code is never used for model training, that zero-day retention contracts are in place with the LLM providers, and that code needed for the review is cleared once the review completes.

Coverage is mapped from Korbit's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Korbit test?+

The coverage map is generated from Korbit's own public product surface (AI code review / developer tooling): 6 scoring areas — Code review issue detection, Pull request workflow integration, and Review configuration and policy, and more — spanning 22 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Korbit evals scored?+

Every case generated for Korbit — across Code review issue detection and Pull request workflow integration and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Korbit library include?+

The full Korbit library is built on request. The coverage map spans 6 areas and 22 capabilities (for example, Bug identification in a diff and Fix explanation quality under Code review issue detection); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Korbit or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Korbit areas and set them up in a Corsac workspace, where you can run every test case against Korbit or your own agent with your own data.