Eval Library
S
For SourcegraphCode Assistant

Deployment Auth And Governance

Sourcegraph (Cody + Amp) · Sourcegraph

Code Intelligence — Sourcegraph

Sourcegraph evals — Deployment, Auth & Governance (relift v3 InfraRed)

About Sourcegraph

Sourcegraph is a code intelligence and AI coding platform: universal code search, precise code navigation, Cody chat grounded in your codebase, cross-repo batch changes, and the Amp autonomous agent — deployed across large enterprise codebases.

Employees

~150

Industry

Code Intelligence

Headquarters

San Francisco, CA

Sample tests· showing 3 of 10

#InputExpected behaviorCheck
01

Customer needs an HA self-hosted Sourcegraph on Kubernetes. Team is mid-way through standing up Docker Compose.

Per docs/admin/deploy, Docker Compose is documented as a single-node deployment; production HA requires Helm-on-Kubernetes (or the Sourcegraph Cloud managed offering). Stop the Compose work, follow the Helm chart guide, and plan migration of any data (repos, configurations) accordingly. [REQUIRES-V…

Pass / FailCode Assistanthigh
02

Customer's IdP rejects the SAML AuthnRequest because the ACS URL registered in Okta does not match Sourcegraph's expected callback.

Per docs/admin/auth/saml, the ACS URL is `<externalURL>/.auth/saml/callback`; misconfiguration of externalURL or the IdP-side ACS causes the rejection. Verify site-config `externalURL` matches the hostname users reach, then update the Okta app to the exact ACS URL.

Pass / FailCode Assistantcritical
03

Switching from SAML to OIDC. Operator copies the openIDConnect site-config snippet but forgets the `clientSecret`.

Per docs/admin/auth/openid_connect, OIDC requires `issuer`, `clientID`, `clientSecret`, and (optionally) `requireEmailDomain`. Reload site config; verify the discovery endpoint resolves and a test login flows. Never commit clientSecret to a repo; use the site-config secret-store hooks.

Pass / FailCode Assistantcritical

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Sourcegraph
  • Code Assistant
  • Deployment Auth And Governance

Recommended for

Sourcegraph (Cody + Amp)Sourcegraph customers

Works with

Related evals

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.