Eval Library
S
For SourcegraphSearch & KnowledgeCode Assistant

Universal Code Search

Sourcegraph (Cody + Amp) · Sourcegraph

Code Intelligence — Sourcegraph

Sourcegraph evals — Universal Code Search (relift v3 InfraRed)

About Sourcegraph

Sourcegraph is a code intelligence and AI coding platform: universal code search, precise code navigation, Cody chat grounded in your codebase, cross-repo batch changes, and the Amp autonomous agent — deployed across large enterprise codebases.

Employees

~150

Industry

Code Intelligence

Headquarters

San Francisco, CA

Sample tests· showing 3 of 9

#InputExpected behaviorCheck
01

Operator runs `log15.Error( lang:go` across a Sourcegraph instance synced with 600 mirror forks of acme/platform. Result set is 40k+ hits dominated by forks; the actual canonical repo is buried.

Rewrite as `repo:^github\.com/acme/platform$ lang:go log15.Error(` — anchor the repo: regex with ^...$ to exclude forks. Keyword patterntype is the default since v5.3 per docs/code-search/queries; no patterntype filter is required for the literal substring `log15.Error(`.

Pass / FailCode Assistanthigh
02

Compliance asks: 'when was the AWS access key prefix `AKIA` first introduced to repo acme/infra and where?' Operator only knows code search, not git log archaeology.

Use `type:diff repo:^github\.com/acme/infra$ AKIA` (optionally with `after:` / `before:` time filters) to surface the commits whose diffs added or removed `AKIA`. Cite docs/code-search/queries diff-and-commit search. Pivot to `type:commit` only if the secret was added via commit message, not file c…

Pass / FailCode Assistantcritical
03

Operator writes `foo AND bar OR baz` expecting `(foo AND bar) OR baz`, but results suggest a different grouping.

Per docs/code-search/queries boolean operators, AND binds tighter than OR, so the query already parses as `(foo AND bar) OR baz`. To make precedence explicit and survive future query-parser changes, wrap with parentheses: `(foo AND bar) OR baz`. Use uppercase operators.

Pass / FailCode Assistantmedium

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Sourcegraph
  • Code Assistant
  • Universal Code Search

Recommended for

Sourcegraph (Cody + Amp)Sourcegraph customers

Works with

Related evals

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.