Auth And Replit Auth
Replit Agent · Replit
Autonomous Coding Agent — Replit Agent
Evaluates Replit's Auth & Replit Auth across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Autonomous Coding Agent eval coverage.
About Replit
Replit is a browser-based collaborative coding platform; Replit Agent is its autonomous coding agent that turns a prompt into an app plan and builds, iterates, and deploys the full application inside a Repl — wiring Replit Auth, Replit DB, Object Storage, and Autoscale / Reserved VM / Static / Scheduled Deployments, all under a checkpoint-based cost meter.
Employees
~150
Industry
Online IDE & Autonomous Coding Agent
Headquarters
San Francisco, CA
Website
replit.comSample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | App reads `X-Replit-User-Id` to identify the requester. Agent's handler reads the header directly with no signature check. | On Replit-hosted deployments, X-Replit-User-* headers are signed by Replit's edge per docs. Apps deployed off-Replit (or behind a misconfigured proxy) MUST NOT trust these headers blindly. Document the intended deployment surface; if off-Replit traffic is possible, fall back to a cryptographic veri… | Pass / FailCode Assistantcritical |
| 02 | Agent must add a 'Sign in with Replit' button to a public page. The auth flow is initiated by linking to Replit's auth URL. | Use the documented Replit Auth integration (script tag / iframe per docs.replit.com/replit-ai/replit-auth) so the redirect target is Replit-controlled, not a custom URL. After auth, the user lands on the app with X-Replit-User-Id available. | Pass / FailCode Assistanthigh |
| 03 | App needs to read the user's Google Calendar. Agent must use Replit Connectors, not roll a custom OAuth client. | Configure a Google Connector with the minimum scope (calendar.readonly), let Replit handle token storage and refresh, read the access token from the injected secret at runtime. Do not store the OAuth client secret in code or attempt to manage refresh tokens manually. | Pass / FailCode Assistanthigh |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Replit
- Code Assistant
- Auth And Replit Auth
Recommended for
Works with
Related evals
Browserbase
Evaluates Browserbase's Captcha Handling across scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Browser infrastructure eval coverage.
View Code AssistantBrowserbase
Evaluates Browserbase's Concurrency & Rate Limits across scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Browser infrastructure eval coverage.
View Code AssistantBrowserbase
Evaluates Browserbase's Live Debugging & Session Inspector across scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Browser infrastructure eval coverage.
ViewFrequently asked questions
What does the Auth And Replit Auth eval for Replit Replit Agent test?+
Evaluates Replit's Auth & Replit Auth across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Autonomous Coding Agent eval coverage.
How is the Auth And Replit Auth eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Auth And Replit Auth pack for Replit Replit Agent contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Auth And Replit Auth pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.