Eval Library
S
For SourcegraphCode Assistant

Cody Autocomplete And Inline Edit

Sourcegraph (Cody + Amp) · Sourcegraph

Code Intelligence — Sourcegraph

Sourcegraph evals — Cody Autocomplete & Inline Edit (relift v3 InfraRed)

About Sourcegraph

Sourcegraph is a code intelligence and AI coding platform: universal code search, precise code navigation, Cody chat grounded in your codebase, cross-repo batch changes, and the Amp autonomous agent — deployed across large enterprise codebases.

Employees

~150

Industry

Code Intelligence

Headquarters

San Francisco, CA

Sample tests· showing 3 of 9

#InputExpected behaviorCheck
01

User types `const userIds = users.map(u => ` in a TS file. Cody suggests `u.id);` — operator presses Tab.

Per docs/cody, autocomplete returns a ghost-text suggestion the user accepts with Tab. Accepted text replaces the suggestion span verbatim; suggestion is debounced per docs to avoid spamming the completion provider. Telemetry records acceptance for the org's Cody analytics.

Pass / FailCode Assistantmedium
02

User types a function signature `function validateOrder(order: Order):` and waits. Cody offers a 12-line body.

Per docs/cody, multi-line completions are gated on language and context (e.g., entering an empty function body). The suggestion must be syntactically valid for the surrounding indentation, and the acceptance cursor must land at the end of the inserted block (not mid-line). Reject visually if it dup…

Pass / FailCode Assistanthigh
03

User highlights a 5-line block and runs Cody 'Edit' with the instruction 'rename camelCase to snake_case in this block only'. Cody also renames a same-named variable two functions away.

Per docs/cody/capabilities/edit, an inline edit operates on the selected range; the model must not return a diff that touches outside the selection. Reject the proposed diff with a clear message, re-prompt narrowly, or fall back to a Batch Change for true file-wide rename.

Pass / FailCode Assistantcritical

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Sourcegraph
  • Code Assistant
  • Cody Autocomplete And Inline Edit

Recommended for

Sourcegraph (Cody + Amp)Sourcegraph customers

Works with

Related evals

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.