Eval Library
GitHub Copilot
For GitHub CopilotCode Assistant

Inline Completions And Ghost Text

GitHub Copilot · GitHub Copilot

AI Coding Assistant — GitHub Copilot

GitHub Copilot evals — Inline Completions & Ghost Text (relift v3 InfraRed)

About GitHub Copilot

GitHub Copilot is GitHub's AI coding assistant — inline ghost-text completions, Copilot Chat with slash commands and @workspace context, the Copilot coding agent and Workspace for repo-wide multi-file tasks, PR summaries and code review on GitHub.com, and gh copilot suggest/explain on the CLI. Copilot ships across VS Code, JetBrains, Visual Studio, the GitHub.com PR/issue surface, and the gh CLI, with a multi-vendor model picker, repo-level custom instructions, public-code / duplication filtering, and enterprise content-exclusion and audit logs.

Employees

~3,000 (GitHub)

Industry

AI Coding Assistant

Headquarters

San Francisco, CA

Sample tests· showing 3 of 9

#InputExpected behaviorCheck
01

Developer types a function signature in VS Code. Copilot proposes a 6-line ghost-text body. Developer presses Tab.

Tab inserts the entire ghost-text suggestion at the cursor verbatim, places the cursor at the suggestion's end, and emits an accept telemetry event tied to the suggestion id so the public-code filter and metrics rollup can attribute it. The editor must not reformat or partially insert.

Pass / FailCode Assistanthigh
02

Developer wants only the first identifier of a Copilot ghost-text suggestion (parseRequest(req, opts)) and presses Cmd+Right (macOS) / Ctrl+Right.

Per docs, partial-accept inserts the next word from the ghost text and leaves the remainder rendered as ghost text past the cursor so the developer can accept the next chunk or dismiss. Acceptance telemetry tags the partial scope.

Pass / FailCode Assistantmedium
03

Ghost text shows suggestion 1 of 3. Developer presses Alt+] to cycle to the next alternative.

Cycle to the next pre-fetched alternative without a network round-trip when available; render '2/3' indicator. Alt+[ cycles backward. Esc dismisses. The accept-telemetry payload records which alternative index was accepted.

Pass / FailCode Assistantmedium

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Github Copilot
  • Code Assistant
  • Inline Completions And Ghost Text

Recommended for

GitHub CopilotGitHub Copilot customers

Works with

Related evals

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.