Deployment Auth And Governance
Sourcegraph (Cody + Amp) · Sourcegraph
Code Intelligence — Sourcegraph
Sourcegraph evals — Deployment, Auth & Governance (relift v3 InfraRed)
About Sourcegraph
Sourcegraph is a code intelligence and AI coding platform: universal code search, precise code navigation, Cody chat grounded in your codebase, cross-repo batch changes, and the Amp autonomous agent — deployed across large enterprise codebases.
Sample tests· showing 3 of 10
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Customer needs an HA self-hosted Sourcegraph on Kubernetes. Team is mid-way through standing up Docker Compose. | Per docs/admin/deploy, Docker Compose is documented as a single-node deployment; production HA requires Helm-on-Kubernetes (or the Sourcegraph Cloud managed offering). Stop the Compose work, follow the Helm chart guide, and plan migration of any data (repos, configurations) accordingly. [REQUIRES-V… | Pass / FailCode Assistanthigh |
| 02 | Customer's IdP rejects the SAML AuthnRequest because the ACS URL registered in Okta does not match Sourcegraph's expected callback. | Per docs/admin/auth/saml, the ACS URL is `<externalURL>/.auth/saml/callback`; misconfiguration of externalURL or the IdP-side ACS causes the rejection. Verify site-config `externalURL` matches the hostname users reach, then update the Okta app to the exact ACS URL. | Pass / FailCode Assistantcritical |
| 03 | Switching from SAML to OIDC. Operator copies the openIDConnect site-config snippet but forgets the `clientSecret`. | Per docs/admin/auth/openid_connect, OIDC requires `issuer`, `clientID`, `clientSecret`, and (optionally) `requireEmailDomain`. Reload site config; verify the discovery endpoint resolves and a test login flows. Never commit clientSecret to a repo; use the site-config secret-store hooks. | Pass / FailCode Assistantcritical |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Sourcegraph
- Code Assistant
- Deployment Auth And Governance
Recommended for
Works with
Related evals
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.