Eval directory
Evals for GitHub Copilot
8 evaluation packs covering adversarial robustness, safety gates, workflow quality, and operator-level checks for GitHub Copilot AI products.
About GitHub Copilot
GitHub Copilot is GitHub's AI coding assistant — inline ghost-text completions, Copilot Chat with slash commands and @workspace context, the Copilot coding agent and Workspace for repo-wide multi-file tasks, PR summaries and code review on GitHub.com, and gh copilot suggest/explain on the CLI. Copilot ships across VS Code, JetBrains, Visual Studio, the GitHub.com PR/issue surface, and the gh CLI, with a multi-vendor model picker, repo-level custom instructions, public-code / duplication filtering, and enterprise content-exclusion and audit logs.
Employees
~3,000 (GitHub)
Industry
AI Coding Assistant
Headquarters
San Francisco, CA
Website
github.com/features/copilotHow complete this published benchmark library is across datasets, metrics, rubrics, use-case maps, and pack context. This is library coverage, not an agent performance score.
Test datasets
8/8 packs
Scoring metrics
0/8 packs
Judge rubrics
8/8 packs
Use-case maps
0/8 packs
Pack context
8/8 packs
Available eval packs for GitHub Copilot
8 packs ready to run.
Copilot Chat In The Ide
Evaluates GitHub Copilot's Copilot Chat in the IDE across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Coding Assistant eval coverage.
Copilot Cli Gh Copilot
Evaluates GitHub Copilot's Copilot CLI (gh copilot) across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Coding Assistant eval coverage.
Copilot Coding Agent And Workspace
Evaluates GitHub Copilot's Copilot Coding Agent & Workspace across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Coding Assistant eval coverage.
Copilot In Github Dot Com And Pr Review
Evaluates GitHub Copilot's Copilot in GitHub.com & PR Review across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Coding Assistant eval coverage.
Inline Completions And Ghost Text
Evaluates GitHub Copilot's Inline Completions & Ghost Text across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Coding Assistant eval coverage.
Knowledge And Context Selection
Evaluates GitHub Copilot's Knowledge & Context Selection across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Coding Assistant eval coverage.
Model Picker And Customization
Evaluates GitHub Copilot's Model Picker & Customization across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Coding Assistant eval coverage.
Safety Privacy And Governance
PII LeakageEvaluates GitHub Copilot's Safety, Privacy & Governance across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Coding Assistant eval coverage.
Why eval GitHub Copilot AI
GitHub Copilot's AI features ship behind brand promises about accuracy, safety, and reliability. Buyers and integrators need to know those promises hold up under adversarial prompts, edge-case workflows, and the long tail of real customer inputs — not just the demo path.
The Corsac eval library for GitHub Copilot measures four dimensions teams care about most when deploying code assistant agents:
- Adversarial robustness — does the agent resist prompt injection, jailbreaks, and social-engineering attempts?
- Workflow quality— does it complete the task buyers were shown in the demo, on inputs that don't look like the demo?
- Safety gates — does it escalate or refuse when it should, and only then?
- Operator quality — does it preserve analyst trust by surfacing the right context at the right time?
Every eval pack above is hand-authored against GitHub Copilot's public product surface and runnable in Corsac with your own data.