Eval directory

Evals for Command Code

Eval coverage for Command Code, mapped from its public product surface.

About Command Code

Command Code is a command-line AI coding agent that builds features, fixes bugs, writes tests, and refactors while continuously learning the user's coding style via a model called taste-1. It runs interactively, headless, or in a background sandbox, and is extensible through skills, slash commands, MCP servers, and plugins. It routes work across many open and closed models under low-cost credit subscriptions ($1 Go, $10 GOAT, Pro/Max tiers) plus a zero-markup Provider API plan.

Industry

AI coding agent (CLI)

Website

commandcode.ai

Use the eval library for Command Code

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Command Code?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Model routing and selection

Choosing, discovering, and switching between the open and closed models Command Code can route work to, across CLI flags and in-session pickers.

“Command Code is the first frontier coding agent that both builds software and continuously learns your coding taste.” commandcode.ai

Mapped capabilities

4 capabilities

Default model behavior
What runs when no model is specified, per the documented registry default.
Model id syntax and matching
-m/--model acceptance of full ids and short names after the slash, case-insensitive matching.
Model discovery surfaces
cmd --list-models, the /model picker, and the docs tables staying consistent with one registry.
Mid-session model switching
Selecting a different model in-session without losing the working context.

Illustrative example

Input: I just installed Command Code and ran `cmd` without any flags. Which model am I using, and how do I print every model id I can pass?
Expected behavior: States that the current default model is deepseek/deepseek-v4-flash and that `cmd --list-models` prints the full set of ids, which matches the /model picker and the docs tables. No other model is named as the default.

Plans, credits, and deals

The subscription tiers, included credit amounts, per-model allowances, and time-bound or permanent model deals that determine effective usage.

Mapped capabilities

4 capabilities

Tier comparison
Go, GOAT, Pro, and Max differences in price, included credits, and request counts.
Deal multipliers and discounts
How 4x, 2x, percentage-off, and free-model deals change effective spend.
Deal validity and expiry
Distinguishing permanent deals from dated promotions and free-while-capacity offers.
Provider API plan
Zero-markup pay-as-you-go access from OpenAI or Anthropic clients.

Illustrative example

Input: On the $10/month GOAT plan, how much credit do I actually get, and how far does that stretch once deals are applied?
Expected behavior: Says $10 buys $70 of credits, a 7x multiplier, and that deals stretch it beyond $100, i.e. more than 10x the price paid. Figures are stated as documented rather than estimated or recomputed.

Taste and continuous learning

The taste-1 model and the feedback loop that turns accepts, rejects, and edits into a persistent profile and project-level skills.

“Every accept, reject, and edit is a signal — auto-generates into project-level skills.” commandcode.ai

Mapped capabilities

4 capabilities

Signal capture
Accept, reject, and edit events feeding the taste profile.
Skill auto-generation
Learned preferences surfacing as project-level skills rather than generic defaults.
Taste portability
npx taste push/pull to move a profile across projects and teammates.
Style adherence over time
Generated code reflecting learned structures, patterns, and tooling choices.

Execution modes and agentic control

The interactive, headless, and sandboxed ways the agent runs, and the controls that bound how much autonomy it takes.

Mapped capabilities

4 capabilities

Interactive and plan mode
Turn-by-turn sessions and planning before execution.
Headless execution
-p one-shot runs and --yolo for unattended flows.
Background sandbox and worktrees
Background tasks and git worktree isolation for parallel work.
Permissions and goal loops
Permission boundaries and /goal running many turns until an independent check verifies completion.

Extensibility and configuration

The hackable layer that lets users add capabilities and shape agent behavior without leaving the CLI.

Mapped capabilities

4 capabilities

Skills and slash commands
Reusable /skills and custom /commands.
MCP servers and plugins
Connecting external tools and installable extensions.
Hooks and mods
Intercepting or altering agent behavior at defined points.
Agents and memory
Custom /agents and persistent /memory carried across sessions.

Core coding workflow

The build, fix, test, and refactor work the agent performs, plus the tools and session mechanics that support long-running engineering tasks.

Mapped capabilities

4 capabilities

Feature and bugfix delivery
Shipping features and fixing defects in an existing codebase.
Tests and refactoring
Writing tests and restructuring code without changing behavior.
Built-in tooling
File ops, shell, grep, the Read tool, and extended thinking.
Context, sessions, and recovery
Compaction, checkpoints, session resume, and tool call repairs.

Coverage is mapped from Command Code's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Command Code test?+

The coverage map is generated from Command Code's own public product surface (AI coding agent (CLI)): 6 scoring areas — Model routing and selection, Plans, credits, and deals, and Taste and continuous learning, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Command Code evals scored?+

Every case generated for Command Code — across Model routing and selection and Plans, credits, and deals and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Command Code library include?+

The full Command Code library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Default model behavior and Model id syntax and matching under Model routing and selection); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Command Code or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Command Code areas and set them up in a Corsac workspace, where you can run every test case against Command Code or your own agent with your own data.