Eval Library
Replit
For ReplitCode Assistant

Repl Workspace And Files

Replit Agent · Replit

Autonomous Coding Agent — Replit Agent

Replit evals — Repl Workspace & Files (relift v3 InfraRed)

About Replit

Replit is a browser-based collaborative coding platform; Replit Agent is its autonomous coding agent that turns a prompt into an app plan and builds, iterates, and deploys the full application inside a Repl — wiring Replit Auth, Replit DB, Object Storage, and Autoscale / Reserved VM / Static / Scheduled Deployments, all under a checkpoint-based cost meter.

Employees

~150

Industry

Online IDE & Autonomous Coding Agent

Headquarters

San Francisco, CA

Website

replit.com

Sample tests· showing 3 of 9

#InputExpected behaviorCheck
01

Agent renames a React component from `UserCard` to `ProfileCard`. It must update the file name, the import sites across 6 other files, and the default export name.

Atomic multi-file edit: rename file, update all imports, update default export. Verify by running the build and checking for missing-import errors before declaring the checkpoint complete.

Pass / FailCode Assistantcritical
02

App needs `ffmpeg` and `imagemagick` as system binaries. Agent should declare them in replit.nix, not assume they preexist.

Add the packages to replit.nix `deps = [ pkgs.ffmpeg pkgs.imagemagick ]` (or the equivalent), then restart the Repl so Nix re-resolves. Verify the binaries are on PATH before declaring done.

Pass / FailCode Assistanthigh
03

Agent must add a `favicon.ico` referenced by `index.html`. The favicon is binary.

Use a workspace-appropriate binary write (e.g., generate the favicon as an actual binary file, not a base64 string written as text). Verify the file opens as a valid ICO. Do not commit a base64 placeholder labeled favicon.ico — the browser will fail to render it.

Pass / FailCode Assistantmedium

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Replit
  • Code Assistant
  • Repl Workspace And Files

Recommended for

Replit AgentReplit customers

Works with

Related evals

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.