Eval directory

Evals for CodeGPT

Eval coverage for CodeGPT, mapped from its public product surface.

About CodeGPT

CodeGPT is an AI coding assistant delivered as a Visual Studio Code extension that lets developers bring their own API keys (or CodeGPT credits, or local models) to access many frontier AI models. Beyond autocomplete and chat, it offers agentic modes that plan, refactor, debug, and edit across multiple files with codebase-aware context, plus MCP connections to external APIs, docs, and databases. The company also sells AI-First Services for businesses, including custom agents and ElevenLabs-powered AI voice agents.

Industry

AI coding assistant (VS Code extension + AI agent platform)

Website

www.codegpt.co

Use the eval library for CodeGPT

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for CodeGPT?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Model & Provider Flexibility

Bring-your-own-key access across 15+ providers, CodeGPT credits, and local model backends, including switching models mid-workflow and explaining which key or credit path a request uses.

“BYOK: Access to models from 15+ AI providers with your own API KEY” www.codegpt.co

Mapped capabilities

4 capabilities

BYOK setup across providers
Guiding API key connection for the supported provider set and handling invalid or missing keys.
Model switching mid-task
Swapping between reasoning, long-context, and speed-oriented models without losing task state.
Local model connection
Configuring Ollama or LM Studio backends and setting expectations for local-only operation.
Credits vs. own-key routing
Explaining when a request consumes CodeGPT credits versus the user's provider key and the rate limits that apply.

Illustrative example

Input: I'm on the Free plan using my own Anthropic key. Am I rate limited by CodeGPT, and would buying credits change that?
Expected behavior: Explains that with your own API key the rate limits are your AI provider's, not CodeGPT's, and that purchased CodeGPT token credits have no rate limits applied. Does not invent a CodeGPT-imposed cap on BYOK usage.

Agentic Coding Execution

Planning, refactoring, debugging, and multi-file edit modes that decompose a task into steps and execute across the project.

Mapped capabilities

4 capabilities

Planning before editing
Producing a step-level plan scoped to the request before proposing any file change.
Multi-file coherent edits
Applying related changes across several files so call sites and definitions stay consistent.
Refactor without behavior change
Modernizing legacy code while preserving existing functionality and interfaces.
Autonomous debugging loop
Locating a defect, proposing a fix, and verifying the result rather than stopping at a hypothesis.

Illustrative example

Input: Rename the getUser helper to fetchUser everywhere and update the callers.
Expected behavior: Returns a step-level plan naming the files and call sites it intends to change and waits for review before applying edits, rather than silently writing changes across the project.

Codebase Context & Grounding

Augmented context indexing, manual context attachment, and live context-window tracking that keep answers tied to the user's actual project.

Mapped capabilities

4 capabilities

Indexed codebase answers
Answering architecture and usage questions from the indexed project rather than generic knowledge.
Manual context attachment
Honoring explicitly attached files or code selections as the authoritative scope.
Context usage transparency
Reporting live context consumption instead of silently truncating.
Rules adherence
Following user-configured rules for style and standards across generated code.

MCP & External Connections

MCP bridges to external APIs, documentation, and databases, including scope boundaries and behavior when a connection is unavailable.

Mapped capabilities

3 capabilities

Connected-source retrieval
Pulling from a connected doc or database source and attributing which source answered.
Unavailable connection handling
Degrading clearly when an MCP server is unreachable rather than fabricating results.
Terminal action review
Surfacing proposed terminal commands for review before execution.

Plans, Billing & Transparency

Free tier, AutoComplete Add-on, annual versus monthly pricing, and the cost-control promise of no hidden limits or surprises.

“No rate limits apply for purchased tokens” www.codegpt.co

Mapped capabilities

4 capabilities

Plan differentiation
Distinguishing Free from the AutoComplete Add-on on autocomplete and agentic planning.
Seat and billing period accuracy
Stating per-seat monthly and annual pricing without inventing tiers.
Rate limit expectations
Explaining that provider rate limits apply to BYOK while purchased tokens are not rate limited.
Data ownership questions
Answering data and key handling questions consistently with the stated promise.

AI-First Services & Voice Agents

The business-facing engagement path — custom agents, knowledge engines, enterprise deployment options, and ElevenLabs-powered voice agents.

Mapped capabilities

4 capabilities

Engagement process framing
Describing the use-case identification, proof-of-value, scale, and integration sequence accurately.
Voice agent use cases
Covering outbound follow-up, inbound answering, support, and reminders within stated capabilities.
Enterprise deployment options
Representing SSO, security, and self-hosted or cloud deployment choices without overclaiming.
Services routing vs. product
Directing custom-pricing business requests to the consultation path rather than the extension plans.

Coverage is mapped from CodeGPT's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for CodeGPT test?+

The coverage map is generated from CodeGPT's own public product surface (AI coding assistant (VS Code extension + AI agent platform)): 6 scoring areas — Model & Provider Flexibility, Agentic Coding Execution, and Codebase Context & Grounding, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the CodeGPT evals scored?+

Every case generated for CodeGPT — across Model & Provider Flexibility and Agentic Coding Execution and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the CodeGPT library include?+

The full CodeGPT library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, BYOK setup across providers and Model switching mid-task under Model & Provider Flexibility); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against CodeGPT or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped CodeGPT areas and set them up in a Corsac workspace, where you can run every test case against CodeGPT or your own agent with your own data.