Eval directory · Code Assistant
Evaluation packs covering adversarial robustness, safety gates, workflow quality, and operator-level checks for Replit AI products.
Replit is a browser-based collaborative coding platform; Replit Agent is its autonomous coding agent that turns a prompt into an app plan and builds, iterates, and deploys the full application inside a Repl — wiring Replit Auth, Replit DB, Object Storage, and Autoscale / Reserved VM / Static / Scheduled Deployments, all under a checkpoint-based cost meter.
Employees
~150
Industry
Online IDE & Autonomous Coding Agent
Headquarters
San Francisco, CA
Website
replit.comUse the eval library for Replit
All 76 test cases — inputs, expected behavior, and pass/fail checks — runnable in Corsac with your own data.
Related in Code Assistant
All evals →Coverage map
9 areas · 76 graded scenarios
Every eval set is graded on
Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls
01
Smoke test for AI coding / IDE assistant completion correctness.
Mapped capabilities
3 scenarios
Example criterion: Replit generates correct, maintainable code completions that satisfy task intent without unsafe patterns.
02
Evaluates Replit's Agent Planning & Build Flow across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Autonomous Coding Agent eval coverage.
Mapped capabilities
9 scenarios
03
Evaluates Replit's Auth & Replit Auth across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Autonomous Coding Agent eval coverage.
Mapped capabilities
9 scenarios
04
Evaluates Replit's Collaboration & Multiplayer across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Autonomous Coding Agent eval coverage.
Mapped capabilities
9 scenarios
05
Evaluates Replit's Deployments across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Autonomous Coding Agent eval coverage.
Mapped capabilities
9 scenarios
06
Evaluates Replit's Repl Workspace & Files across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Autonomous Coding Agent eval coverage.
Mapped capabilities
9 scenarios
07
Evaluates Replit's Replit DB & Storage across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Autonomous Coding Agent eval coverage.
Mapped capabilities
9 scenarios
08
Evaluates Replit's Safety, Cost & Governance across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Autonomous Coding Agent eval coverage.
Mapped capabilities
10 scenarios
09
Evaluates Replit's Tool Use & Mid-build Function Calls across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Autonomous Coding Agent eval coverage.
Mapped capabilities
9 scenarios
Each eval pack tests Replit's public product surface — including Ghostwriter Completion Smoke V1, Agent Planning And Build Flow, Auth And Replit Auth — against graded scenarios covering adversarial robustness, workflow quality, safety gates, and operator quality. Every pack is runnable in Corsac with your own data.
Every eval set is graded the same way: pass/fail checks plus an LLM judge scoring 1–5 against each case's expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.
The Replit eval library includes 76 graded test cases across 9 eval packs. Each case defines an input, expected behavior, and pass/fail criteria.
Request the library with your work email above and we'll set it up in a Corsac workspace, where you can run every test case against Replit or your own agent with your own data.
