Deployment Auth And Governance
Sourcegraph (Cody + Amp) · Sourcegraph
Code Intelligence — Sourcegraph
Evaluates Sourcegraph's Deployment, Auth & Governance across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Code Intelligence eval coverage.
About Sourcegraph
Sourcegraph is a code intelligence and AI coding platform: universal code search, precise code navigation, Cody chat grounded in your codebase, cross-repo batch changes, and the Amp autonomous agent — deployed across large enterprise codebases.
Sample tests· showing 3 of 10
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Customer needs an HA self-hosted Sourcegraph on Kubernetes. Team is mid-way through standing up Docker Compose. | Per docs/admin/deploy, Docker Compose is documented as a single-node deployment; production HA requires Helm-on-Kubernetes (or the Sourcegraph Cloud managed offering). Stop the Compose work, follow the Helm chart guide, and plan migration of any data (repos, configurations) accordingly. [REQUIRES-V… | Pass / FailCode Assistanthigh |
| 02 | Customer's IdP rejects the SAML AuthnRequest because the ACS URL registered in Okta does not match Sourcegraph's expected callback. | Per docs/admin/auth/saml, the ACS URL is `<externalURL>/.auth/saml/callback`; misconfiguration of externalURL or the IdP-side ACS causes the rejection. Verify site-config `externalURL` matches the hostname users reach, then update the Okta app to the exact ACS URL. | Pass / FailCode Assistantcritical |
| 03 | Switching from SAML to OIDC. Operator copies the openIDConnect site-config snippet but forgets the `clientSecret`. | Per docs/admin/auth/openid_connect, OIDC requires `issuer`, `clientID`, `clientSecret`, and (optionally) `requireEmailDomain`. Reload site config; verify the discovery endpoint resolves and a test login flows. Never commit clientSecret to a repo; use the site-config secret-store hooks. | Pass / FailCode Assistantcritical |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Sourcegraph
- Code Assistant
- Deployment Auth And Governance
Recommended for
Works with
Related evals
Browserbase
Evaluates Browserbase's Captcha Handling across scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Browser infrastructure eval coverage.
View Code AssistantBrowserbase
Evaluates Browserbase's Concurrency & Rate Limits across scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Browser infrastructure eval coverage.
View Code AssistantBrowserbase
Evaluates Browserbase's Live Debugging & Session Inspector across scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Browser infrastructure eval coverage.
ViewFrequently asked questions
What does the Deployment Auth And Governance eval for Sourcegraph Sourcegraph (Cody + Amp) test?+
Evaluates Sourcegraph's Deployment, Auth & Governance across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Code Intelligence eval coverage.
How is the Deployment Auth And Governance eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Deployment Auth And Governance pack for Sourcegraph Sourcegraph (Cody + Amp) contains 10 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Deployment Auth And Governance pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.