Knowledge And Context Selection
GitHub Copilot · GitHub Copilot
AI Coding Assistant — GitHub Copilot
Evaluates GitHub Copilot's Knowledge & Context Selection across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Coding Assistant eval coverage.
About GitHub Copilot
GitHub Copilot is GitHub's AI coding assistant — inline ghost-text completions, Copilot Chat with slash commands and @workspace context, the Copilot coding agent and Workspace for repo-wide multi-file tasks, PR summaries and code review on GitHub.com, and gh copilot suggest/explain on the CLI. Copilot ships across VS Code, JetBrains, Visual Studio, the GitHub.com PR/issue surface, and the gh CLI, with a multi-vendor model picker, repo-level custom instructions, public-code / duplication filtering, and enterprise content-exclusion and audit logs.
Employees
~3,000 (GitHub)
Industry
AI Coding Assistant
Headquarters
San Francisco, CA
Website
github.com/features/copilotSample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | @workspace is queried right after the developer pulls 1,200 new commits (branch switch). | The local workspace index must invalidate the changed file ranges and either (a) re-index incrementally before answering, or (b) surface a 'stale index' notice so the developer knows results may not reflect the current branch. Do not silently answer from the pre-pull index. | Pass / FailCode Assistanthigh |
| 02 | Developer asks `@web how do I configure the Stripe webhook secret?`. | @web (where enabled by the org policy) performs a web search, returns snippets with source URLs, and grounds the answer in those snippets. The answer must clearly distinguish public docs guidance from repo-specific context. If @web is disabled by org policy, surface that fact, do not fall back to t… | Pass / FailCode Assistanthigh |
| 03 | Developer attaches a config.yaml as Chat context and selects 5 lines of main.go. They ask 'apply the yaml to this code.' | Chat must use both context sources — the selection as the primary edit target, the attached file as the spec. The answer should reference both by name. Apply-in-editor must scope to the main.go selection only; do not edit config.yaml unless explicitly requested. | Pass / FailCode Assistanthigh |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Github Copilot
- Code Assistant
- Knowledge And Context Selection
Recommended for
Works with
Related evals
Browserbase
Evaluates Browserbase's Captcha Handling across scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Browser infrastructure eval coverage.
View Code AssistantBrowserbase
Evaluates Browserbase's Concurrency & Rate Limits across scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Browser infrastructure eval coverage.
View Code AssistantBrowserbase
Evaluates Browserbase's Live Debugging & Session Inspector across scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Browser infrastructure eval coverage.
ViewFrequently asked questions
What does the Knowledge And Context Selection eval for GitHub Copilot GitHub Copilot test?+
Evaluates GitHub Copilot's Knowledge & Context Selection across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Coding Assistant eval coverage.
How is the Knowledge And Context Selection eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Knowledge And Context Selection pack for GitHub Copilot GitHub Copilot contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Knowledge And Context Selection pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.