Corsac
Use Corsac
All evals
GitHub Copilot

Eval directory

Evals for GitHub Copilot

8 evaluation packs covering adversarial robustness, safety gates, workflow quality, and operator-level checks for GitHub Copilot AI products.

Code Assistant
Use evals for GitHub Copilot

About GitHub Copilot

GitHub Copilot is GitHub's AI coding assistant — inline ghost-text completions, Copilot Chat with slash commands and @workspace context, the Copilot coding agent and Workspace for repo-wide multi-file tasks, PR summaries and code review on GitHub.com, and gh copilot suggest/explain on the CLI. Copilot ships across VS Code, JetBrains, Visual Studio, the GitHub.com PR/issue surface, and the gh CLI, with a multi-vendor model picker, repo-level custom instructions, public-code / duplication filtering, and enterprise content-exclusion and audit logs.

Employees

~3,000 (GitHub)

Industry

AI Coding Assistant

Headquarters

San Francisco, CA

Website

github.com/features/copilot

Available eval packs for GitHub Copilot

8 packs ready to run.

Why eval GitHub Copilot AI

GitHub Copilot's AI features ship behind brand promises about accuracy, safety, and reliability. Buyers and integrators need to know those promises hold up under adversarial prompts, edge-case workflows, and the long tail of real customer inputs — not just the demo path.

The Corsac eval library for GitHub Copilot measures four dimensions teams care about most when deploying code assistant agents:

  • Adversarial robustness — does the agent resist prompt injection, jailbreaks, and social-engineering attempts?
  • Workflow quality— does it complete the task buyers were shown in the demo, on inputs that don't look like the demo?
  • Safety gates — does it escalate or refuse when it should, and only then?
  • Operator quality — does it preserve analyst trust by surfacing the right context at the right time?

Every eval pack above is hand-authored against GitHub Copilot's public product surface and runnable in Corsac with your own data.