
Cody Autocomplete And Inline Edit
Sourcegraph (Cody + Amp) · Sourcegraph
Code Intelligence — Sourcegraph
Evaluates Sourcegraph's Cody Autocomplete & Inline Edit across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Code Intelligence eval coverage.
About Sourcegraph
Sourcegraph is a code intelligence and AI coding platform: universal code search, precise code navigation, Cody chat grounded in your codebase, cross-repo batch changes, and the Amp autonomous agent — deployed across large enterprise codebases.
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | User types `const userIds = users.map(u => ` in a TS file. Cody suggests `u.id);` — operator presses Tab. | Per docs/cody, autocomplete returns a ghost-text suggestion the user accepts with Tab. Accepted text replaces the suggestion span verbatim; suggestion is debounced per docs to avoid spamming the completion provider. Telemetry records acceptance for the org's Cody analytics. | Pass / FailCode Assistantmedium |
| 02 | User types a function signature `function validateOrder(order: Order):` and waits. Cody offers a 12-line body. | Per docs/cody, multi-line completions are gated on language and context (e.g., entering an empty function body). The suggestion must be syntactically valid for the surrounding indentation, and the acceptance cursor must land at the end of the inserted block (not mid-line). Reject visually if it dup… | Pass / FailCode Assistanthigh |
| 03 | User highlights a 5-line block and runs Cody 'Edit' with the instruction 'rename camelCase to snake_case in this block only'. Cody also renames a same-named variable two functions away. | Per docs/cody/capabilities/edit, an inline edit operates on the selected range; the model must not return a diff that touches outside the selection. Reject the proposed diff with a clear message, re-prompt narrowly, or fall back to a Batch Change for true file-wide rename. | Pass / FailCode Assistantcritical |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Sourcegraph
- Code Assistant
- Cody Autocomplete And Inline Edit
Recommended for
Works with
Related evals
Browserbase
Evaluates Browserbase's Captcha Handling across scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Browser infrastructure eval coverage.
View Code AssistantBrowserbase
Evaluates Browserbase's Concurrency & Rate Limits across scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Browser infrastructure eval coverage.
View Code AssistantBrowserbase
Evaluates Browserbase's Live Debugging & Session Inspector across scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Browser infrastructure eval coverage.
ViewFrequently asked questions
What does the Cody Autocomplete And Inline Edit eval for Sourcegraph Sourcegraph (Cody + Amp) test?+
Evaluates Sourcegraph's Cody Autocomplete & Inline Edit across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Code Intelligence eval coverage.
How is the Cody Autocomplete And Inline Edit eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Cody Autocomplete And Inline Edit pack for Sourcegraph Sourcegraph (Cody + Amp) contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Cody Autocomplete And Inline Edit pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.