Sourcegraph
For SourcegraphCode Assistant

Cody Autocomplete And Inline Edit

Sourcegraph (Cody + Amp) · Sourcegraph

Code Intelligence — Sourcegraph

Evaluates Sourcegraph's Cody Autocomplete & Inline Edit across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Code Intelligence eval coverage.

About Sourcegraph

Sourcegraph is a code intelligence and AI coding platform: universal code search, precise code navigation, Cody chat grounded in your codebase, cross-repo batch changes, and the Amp autonomous agent — deployed across large enterprise codebases.

Employees

~150

Industry

Code Intelligence

Headquarters

San Francisco, CA

Sample tests· showing 3 of 9

#InputExpected behaviorCheck
01

User types `const userIds = users.map(u => ` in a TS file. Cody suggests `u.id);` — operator presses Tab.

Per docs/cody, autocomplete returns a ghost-text suggestion the user accepts with Tab. Accepted text replaces the suggestion span verbatim; suggestion is debounced per docs to avoid spamming the completion provider. Telemetry records acceptance for the org's Cody analytics.

Pass / FailCode Assistantmedium
02

User types a function signature `function validateOrder(order: Order):` and waits. Cody offers a 12-line body.

Per docs/cody, multi-line completions are gated on language and context (e.g., entering an empty function body). The suggestion must be syntactically valid for the surrounding indentation, and the acceptance cursor must land at the end of the inserted block (not mid-line). Reject visually if it dup…

Pass / FailCode Assistanthigh
03

User highlights a 5-line block and runs Cody 'Edit' with the instruction 'rename camelCase to snake_case in this block only'. Cody also renames a same-named variable two functions away.

Per docs/cody/capabilities/edit, an inline edit operates on the selected range; the model must not return a diff that touches outside the selection. Reject the proposed diff with a clear message, re-prompt narrowly, or fall back to a Batch Change for true file-wide rename.

Pass / FailCode Assistantcritical

Unlock full benchmark

6 more test cases

Use this benchmark

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Sourcegraph
  • Code Assistant
  • Cody Autocomplete And Inline Edit

Recommended for

Sourcegraph (Cody + Amp)Sourcegraph customers

Works with

Related evals

Frequently asked questions

What does the Cody Autocomplete And Inline Edit eval for Sourcegraph Sourcegraph (Cody + Amp) test?+

Evaluates Sourcegraph's Cody Autocomplete & Inline Edit across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Code Intelligence eval coverage.

How is the Cody Autocomplete And Inline Edit eval scored?+

The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

How many test cases does this eval pack include?+

The Cody Autocomplete And Inline Edit pack for Sourcegraph Sourcegraph (Cody + Amp) contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.

How do I run this eval?+

Sign up for Corsac, connect your model or agent endpoint, and run the Cody Autocomplete And Inline Edit pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.