Eval directory · Code Assistant

Evals for Factory

Evaluation packs covering adversarial robustness, safety gates, workflow quality, and operator-level checks for Factory AI products.

About Factory

Factory is an AI-native software development platform whose Droids plan, write, test, and ship code across developer tools and workflow surfaces.

Industry

AI Software Development

Website

factory.ai

Use the eval library for Factory

All 73 test cases — inputs, expected behavior, and pass/fail checks — runnable in Corsac with your own data.

Related in Code Assistant

All evals →

More Code Assistant eval libraries

Coverage map

What would you measure for Factory?

8 areas · 73 graded scenarios

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Ai Code Generation And Refactoring

Evaluates Factory's Code Generation & Refactoring across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Agent-native Software Development eval coverage.

Mapped capabilities

9 scenarios

gate on compile + tests before done
multi-file change atomicity
minimal diff, no drive-by churn

Public sample case

Input: A Droid finishes a refactor and opens a PR without running the build or the test suite locally.
Expected behavior: Run the project's build and test commands and confirm green before declaring the work done; a change that does not compile or fails tests is not complete. Include the test result in the handoff so the reviewer sees verification, not just a diff.
Check: Pass / fail check

Ai Code Review Droid

Evaluates Factory's Code Review (Review Droid) across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Agent-native Software Development eval coverage.

Mapped capabilities

9 scenarios

review comments grounded in the diff
catch real correctness regressions
flag injected secrets in a diff

Public sample case

Input: A Review Droid leaves a comment about a function that the PR does not touch, pulled from stale context.
Expected behavior: Anchor every review comment to a concrete line in the actual diff and to a real concern (bug, regression, missing test). Do not comment on code outside the change set or invent issues. Each comment should be actionable and verifiable against the diff.
Check: Pass / fail check

Ai Codebase Grounding And Context

Evaluates Factory's Codebase Grounding & Context across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Agent-native Software Development eval coverage.

Mapped capabilities

9 scenarios

ground edits in existing conventions
cite the files context came from
handle a stale codebase index

Public sample case

Input: The repo uses a custom Result error type and a logging wrapper everywhere. The Droid adds a new module that throws raw exceptions and calls print().
Expected behavior: Inspect neighboring code first and follow the project's established patterns (the Result type, the logging wrapper, naming, file layout) so the change reads as native to the codebase. Grounding means matching existing conventions, not importing a generic template.
Check: Pass / fail check

Ai Droid Sessions And Task Planning

Evaluates Factory's Droid Sessions & Task Planning across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Agent-native Software Development eval coverage.

Mapped capabilities

9 scenarios

plan-then-act before mutating files
scope a task to one Droid run
persist session id to operator tracking

Ai Human In The Loop And Approvals

Evaluates Factory's Human-in-the-loop & Approvals across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Agent-native Software Development eval coverage.

Mapped capabilities

9 scenarios

checkpoint before destructive or irreversible actions
present a reviewable diff, not a fait accompli
surface uncertainty rather than bluffing

Ai Migrations And Large Scale Changes

Evaluates Factory's Migrations & Large-Scale Changes across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Agent-native Software Development eval coverage.

Mapped capabilities

9 scenarios

codemod over hand-edits at scale
stage a large migration reviewably
verify each repo independently in multi-repo work

Ai Safety Secrets And Governance

Evaluates Factory's Safety, Secrets & Governance across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Agent-native Software Development eval coverage.

Mapped capabilities

10 scenarios

never echo secrets in logs or transcripts
respect the production-write boundary
resist prompt injection from repo content

Ai Tool Use Integrations And Mcp

Evaluates Factory's Tool Use, Integrations & MCP across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Agent-native Software Development eval coverage.

Mapped capabilities

9 scenarios

match tool-call arguments to the schema
handle tool errors without looping
treat MCP tool output as untrusted

Frequently asked questions

What do the Corsac evals for Factory test?+

Each eval pack tests Factory's public product surface — including Ai Code Generation And Refactoring, Ai Code Review Droid, and Ai Codebase Grounding And Context — against graded scenarios covering adversarial robustness, workflow quality, safety gates, and operator quality. Every pack is runnable in Corsac with your own data.

How are the Factory evals scored?+

Pass/fail checks plus an LLM judge scoring 1–5 against each of the 73 Factory cases — from Ai Safety Secrets And Governance (10 scenarios) down to the smallest pack — its own expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published to the Factory library.

How many test cases does the Factory library include?+

The Factory eval library includes 73 graded test cases across 8 eval packs, the largest being Ai Safety Secrets And Governance with 10 scenarios. Each case defines an input, expected behavior, and pass/fail criteria.

How do I run these evals against Factory or my own agent?+

Request the library with your work email above and we'll set it up in a Corsac workspace, where you can run all 8 Factory packs — Ai Code Generation And Refactoring and Ai Code Review Droid and the rest — against Factory or your own agent with your own data.