Eval directory

Evals for GitHub

Eval coverage for GitHub, mapped from its public product surface.

About GitHub

GitHub is a developer platform where developers, AI agents, and code come together to build, scale, and deliver software. It combines source hosting and collaboration with GitHub Copilot for AI-assisted coding, GitHub Actions for CI/CD, and security features like Dependabot, Code Security, and Secret Protection. Plans range from a free tier for individuals and organizations to Team and Enterprise tiers, with a 30-day trial covering Enterprise, Copilot, and Advanced Security.

Industry

developer platform with AI coding assistant (GitHub Copilot)

Website

github.com

Use the eval library for GitHub

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for GitHub?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Plans, entitlements & account access

Correctly distinguishing Free, Team, and Enterprise tiers, the quotas attached to each, the 30-day trial covering Enterprise/Copilot/Advanced Security, and the sign-in and account-creation paths.

“Copilot is ready to work with you at each step of the software development lifecycle.” github.com

Mapped capabilities

4 capabilities

Tier feature boundaries
What is included in Free vs. Team (Codespaces, protected branches, code owners, draft PRs, web-based support) vs. what requires Enterprise (SAML, self-hosting, priority support).
Quota accuracy
Actions minutes (2,000/mo Free, 3,000/mo Team) and Packages storage (500MB Free, 2GB Team), including that both are free for public repositories.
Trial scope and framing
The 30-day free trial bundles Enterprise, Copilot, and Advanced Security; not misstating what the trial covers or its cost.
Sign-in and account creation
Routing users to sign in, passkey-based account creation, password recovery, and multi-account/session-reload behavior.

Illustrative example

Input: I'm on GitHub's Free plan. How many CI/CD minutes do I get each month, and does that cap apply to my public repositories?
Expected behavior: States that the Free plan includes 2,000 CI/CD minutes per month for GitHub Actions, and notes that Actions usage is free for public repositories so the monthly cap applies to private repository workflows.

Repository hosting & collaboration

The source-hosting and code-review core: repositories, issues and projects, pull requests, and the review controls that gate merges.

“Keep projects secure by automatically opening pull requests to update vulnerable dependencies and keep them up to date.” github.com

Mapped capabilities

4 capabilities

Repository visibility
Unlimited public and private repositories, and that public repositories are accessible to anyone on GitHub.com via web or command line.
Pull request review controls
Multiple reviewers, required reviewers, code owners, draft pull requests, and protected branches as Team-tier collaboration features.
Issues & Projects
Project management features that adapt to team, project, and workflow, and how they relate to repository work.
Pages, Wikis & integrations
Publishing and documentation surfaces plus connecting existing tools/integrations into the workflow.

GitHub Copilot & agentic coding

AI assistance across the software development lifecycle, including Copilot Chat's Ask and Agent modes and the boundary between explaining code and editing it.

“With GitHub, developers, agents, and code come together on one platform.” github.com

Mapped capabilities

4 capabilities

Ask vs. Agent mode
Distinguishing conversational answers from agent mode that analyzes a codebase, proposes edits across files, applies them, and summarizes changes.
Code generation & refactoring
Writing, testing, and fixing code from boilerplate through complex features, including extracting duplicated logic into reusable functions.
SDLC coverage claims
Where Copilot is positioned across the lifecycle (code, plan, collaborate, automate, secure) without overstating autonomy.
Change explanation & review
Summarizing what an agent changed and why, so a human reviewer can verify the diff before merging.

Illustrative example

Input: In Copilot Chat, what changes when I switch the mode dropdown from Ask to Agent?
Expected behavior: Explains that Ask mode answers questions about code conversationally, while Agent mode analyzes the codebase, determines which files need changes, generates the edits across them, and then summarizes the completed work for review.

Automation with GitHub Actions

CI/CD workflows that build, test, and deploy projects, and the metering model that governs how much automation a plan allows.

“Write, test, and fix code quickly with GitHub Copilot, from simple boilerplate to complex features.” github.com

Mapped capabilities

4 capabilities

Workflow composition
Writing tasks and combining them into workflows that build, test, and deploy any code project hosted on GitHub.
Minutes metering
How execution minutes are consumed per plan and the public-repository exemption.
Packages in the pipeline
Hosting software packages or consuming them as dependencies, in both private and public hosting modes.
Automation vs. review gates
How automated workflows interact with protected branches and required reviewers before deployment.

Security & supply chain protection

The security add-ons and defaults: Dependabot updates on every tier, plus Code Security and Secret Protection as featured add-ons.

Mapped capabilities

4 capabilities

Dependabot updates
Automatically opening pull requests to update vulnerable dependencies and keep them current, available from the Free tier.
Code Security
Finding and fixing vulnerabilities in code before it reaches production, and which plans/add-ons provide it.
Secret Protection
Preventing secret leaks before they happen and remediating secrets that already exist in a repository.
Add-on vs. included
Not conflating featured security add-ons with baseline plan inclusions or with the bundled 30-day trial.

Agent integration & MCP registry

Connecting AI assistants to GitHub and to third-party tools through the MCP registry, including the official GitHub MCP server's scope of control.

Mapped capabilities

4 capabilities

GitHub MCP server scope
Managing repositories, issues, pull requests, and workflows through natural language, and the limits of that control surface.
Registry browsing & install
Finding servers among the listed catalog and following the install path for a chosen server.
Third-party server descriptions
Accurately representing what a listed server does (e.g. browser automation, database access, docs retrieval) without inventing capabilities.
Public discovery surfaces
Topics and featured topic pages as a way to browse public projects by subject.

Coverage is mapped from GitHub's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for GitHub test?+

The coverage map is generated from GitHub's own public product surface (developer platform with AI coding assistant (GitHub Copilot)): 6 scoring areas — Plans, entitlements & account access, Repository hosting & collaboration, and GitHub Copilot & agentic coding, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the GitHub evals scored?+

Every case generated for GitHub — across Plans, entitlements & account access and Repository hosting & collaboration and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the GitHub library include?+

The full GitHub library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Tier feature boundaries and Quota accuracy under Plans, entitlements & account access); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against GitHub or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped GitHub areas and set them up in a Corsac workspace, where you can run every test case against GitHub or your own agent with your own data.