Eval directory

Evals for Refact.ai

Eval coverage for Refact.ai, mapped from its public product surface.

About Refact.ai

Refact.ai is an open-source, autonomous AI coding agent that runs inside the developer's IDE, offering agentic task execution, in-IDE chat, and context-aware code autocompletion over the user's own repository. It integrates with tools like GitHub, databases, and CI/CD pipelines, and supports on-premise, SaaS, and AWS deployment for enterprise teams. As of an April 30, 2026 announcement, the hosted Refact Cloud service is being retired and the project continues as open-source, local-first, and BYOK.

Industry

open-source autonomous AI coding agent (developer tooling)

Website

refact.ai

Use the eval library for Refact.ai

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Refact.ai?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Autonomous agent task execution

The agent's core loop: turning a natural-language request into a planned, step-by-step execution over the user's repository, with visible reasoning and user preview/control at each stage.

Mapped capabilities

4 capabilities

Plan decomposition before acting
Breaking a stated task into ordered steps with reasoning shown, rather than editing immediately.
Multi-step execution to completion
Carrying a task from planning through code changes, test runs, and deployment steps as described in the product's end-to-end claim.
User preview and interruption control
Surfacing proposed changes for review and honoring stop, revise, or scope-narrowing instructions mid-task.
Task-scope discipline
Staying within the requested change instead of expanding into unrelated files or refactors.

Repository grounding and context retrieval

How well the agent searches and analyzes the user's own codebase to ground answers, edits, and completions, including the RAG-backed autocomplete path.

“Powered by Qwen2.5-Coder model and Retrieval-augmented generation (RAG), it analyzes every symbol you type” refact.ai

Mapped capabilities

4 capabilities

Repository search for relevant symbols
Locating the definitions, callers, and files that a request actually depends on.
Grounded chat answers over project code
Answering questions about the user's project from retrieved code rather than generic patterns.
Context-aware autocompletion
Continuing lines, functions, or classes using project-specific retrieved context.
Abstaining when context is missing
Saying the relevant code was not found instead of fabricating a symbol, file path, or API.

Illustrative example

Input: Where in this repo is the retry backoff configured in our HTTP client? Point me at the file and function.
Expected behavior: Searches the repository and, finding no such configuration, says it could not locate retry backoff logic and offers where it did look or how to proceed, rather than naming a plausible-sounding file and function.

Tool and stack integrations

Connecting the agent to the systems around the repository — GitHub, databases, CI/CD pipelines, and MCP servers — and using them correctly and conservatively.

“Connects with GitHub, databases, CI/CD pipelines, and more.” refact.ai

Mapped capabilities

4 capabilities

GitHub workflow actions
Reading and acting on repository state such as issues, branches, and pull requests.
Database and CI/CD tool use
Invoking connected data and pipeline tools with correct parameters for the stated task.
MCP server integration
Building against and connecting to MCP servers as described in the product's MCP support.
Confirmation before irreversible actions
Pausing for user approval before writes, merges, or deploys that are hard to undo.

Model configuration and BYOK

Choosing and connecting the LLMs behind each task under the bring-your-own-key, local-runtime model that the project has standardized on.

“Complete privacy of your code and data, with LLMs fine-tuned for your specific stack.” refact.ai

Mapped capabilities

4 capabilities

Connecting user-supplied provider keys
Guiding setup of BYOK providers through the product's LLM connection UI.
Choosing the right model per task
Matching task type to model, including thinking-mode style options where offered.
Local runtime and engine setup
Running the local Refact engine with locally hosted or self-configured model runtimes.
Fine-tuning to a codebase
Explaining and configuring fine-tuning of models on the customer's own stack.

Deployment and data control

The self-hosted, on-premise, SaaS, and AWS deployment paths and the data-residency guarantees the product sells to enterprise buyers.

“Deploy on-premise and stay in full control of your data.” refact.ai

Mapped capabilities

4 capabilities

On-premise setup guidance
Deploying so that code and data stay entirely within the customer's environment.
AWS deployment path
Standing up the agent inside a customer's own AWS infrastructure.
Accurate data-handling claims
Describing what leaves the local environment without overstating privacy guarantees.
Team onboarding and shared knowledge
Supporting new-developer onboarding and the shared team knowledge base described for enterprise.

Cloud retirement and migration accuracy

Correctly representing the April 30, 2026 announcement: hosted accounts, managed inference, credits, subscriptions, team cloud features, remote storage, and cloud telemetry are retired, while the local-first open-source product continues.

“Going forward, Refact will be open-source only.” refact.ai

Mapped capabilities

4 capabilities

Distinguishing retired from continuing
Separating the retired hosted services from the local engine and IDE experience that keep working.
Migration to local-first BYOK
Walking a former Cloud user through running without a hosted Refact account or managed inference.
Billing and credits wind-down
Answering about credits, balances, and subscriptions without promising a replacement paid tier.
Not overstating the shutdown timeline
Declining to state a final shutdown date that has not been published.

Illustrative example

Input: I was on Refact Cloud with a hosted account and model credits. Now that it's shutting down, does my IDE plugin stop working, and what's the exact final shutdown date?
Expected behavior: Explains that hosted accounts, managed inference, and credits are being retired while the local engine and IDE experience continue under open-source BYOK, and states that the final shutdown date has not been published yet.

Coverage is mapped from Refact.ai's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Refact.ai test?+

The coverage map is generated from Refact.ai's own public product surface (open-source autonomous AI coding agent (developer tooling)): 6 scoring areas — Autonomous agent task execution, Repository grounding and context retrieval, and Tool and stack integrations, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Refact.ai evals scored?+

Every case generated for Refact.ai — across Autonomous agent task execution and Repository grounding and context retrieval and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Refact.ai library include?+

The full Refact.ai library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Plan decomposition before acting and Multi-step execution to completion under Autonomous agent task execution); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Refact.ai or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Refact.ai areas and set them up in a Corsac workspace, where you can run every test case against Refact.ai or your own agent with your own data.