Eval directory

Evals for OpenCode

Eval coverage for OpenCode, mapped from its public product surface.

About OpenCode

OpenCode is an open source AI coding agent that helps developers write code from the terminal, an IDE extension, or a desktop app. It works with free included models or any model from 75+ providers via Models.dev, including local models and existing GitHub Copilot or ChatGPT Plus/Pro subscriptions. An enterprise offering adds centralized config with SSO and internal AI gateway integration, sold per seat.

Industry

open source AI coding agent

Website

opencode.ai

Use the eval library for OpenCode

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for OpenCode?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Install & Platform Setup

Getting OpenCode running across the documented install paths and platforms, and setting the prerequisites the docs call out.

Mapped capabilities

4 capabilities

Package manager install paths
install script (curl | bash), npm/bun/pnpm/yarn global installs, Homebrew via anomalyco/tap vs. the Homebrew-maintained opencode formula, pacman/paru, choco/scoop, mise, Docker image.
Platform guidance
macOS, Linux, Windows (WSL recommended over native), and the desktop beta availability across all three.
Terminal prerequisites
modern terminal emulator recommendations (WezTerm, Alacritty, Ghostty, Kitty) and which are cross-platform vs. Linux/macOS only.
Release channel freshness
which install source tracks the most up-to-date releases (OpenCode tap, AUR latest) versus slower-moving stable channels.

Model & Provider Connectivity

Choosing and connecting the model behind the agent, including included models, third-party providers, existing subscriptions, and local models.

“75+ LLM providers through Models.dev, including local models” opencode.ai

Mapped capabilities

4 capabilities

Free included models
models shipped with OpenCode and the fact that no additional AI subscription is required to get started.
Any-provider connection
75+ LLM providers through Models.dev, including Claude, GPT, and Gemini; API keys as the prerequisite for provider access.
Existing subscription reuse
logging in with GitHub to use a Copilot account, and logging in with OpenAI to use a ChatGPT Plus or Pro account.
Local models and Zen
running local models via Models.dev, and Zen as a curated set of models OpenCode has tested and benchmarked for coding agents.

Illustrative example

Input: I already pay for GitHub Copilot. Do I need to buy separate API credits to use OpenCode?
Expected behavior: Answers no: OpenCode ships with free included models, and a Copilot subscription can be reused by logging in with GitHub. May note that connecting other providers from the 75+ on Models.dev requires that provider's API key.

Agent Session Workflow

How developers actually work with the agent day to day across the terminal, IDE extension, and desktop app.

“Available as a terminal interface, desktop app, and IDE extension” opencode.ai

Mapped capabilities

4 capabilities

Multi-session parallelism
starting multiple agents in parallel on the same project.
Automatic LSP loading
automatically loading the right LSPs for the LLM.
Session sharing
the /share feature and share links used for reference or debugging.
Surface parity across clients
terminal interface, desktop app (including the new tabs feature), and IDE extension as interchangeable entry points.

Privacy & Data Handling

The privacy-first posture the product markets, and the specific exception a reviewer must understand before approving use.

“OpenCode does not store any of your code or context data” opencode.ai

Mapped capabilities

4 capabilities

No code or context storage
OpenCode does not store code or context data; processing happens locally or through direct API calls to the chosen AI provider.
/share data flow exception
enabled shares send the conversation and associated data to the hosting service at opencode.ai, served through a CDN edge network and cached near users.
Disabling sharing via config
opencode.json with "share": "disabled", and the documented recommendation to disable it during a trial.
Code ownership
the user owns all code produced by OpenCode, with no licensing restrictions or ownership claims.

Illustrative example

Input: Our security team was told OpenCode never sends our data anywhere. Is that accurate, and is there anything we should turn off before a trial?
Expected behavior: States that OpenCode does not store code or context data and processes locally or via direct provider calls, then names /share as the exception that sends conversation data to opencode.ai's CDN, and recommends disabling it for the trial.

Enterprise Configuration & Buying

What the enterprise offering adds for organizations that need code and data to stay in their infrastructure.

Mapped capabilities

4 capabilities

Centralized config
organization-wide configuration as the mechanism for enforcing enterprise settings.
SSO and internal AI gateway
integrating centralized config with the organization's SSO and internal AI gateway.
Per-seat pricing model
per-seat licensing, and no token charges when the customer brings their own LLM gateway.
Trial-to-deployment path
self-serve internal trial on the open source build, then contacting the team to discuss pricing and implementation options.

Docs & Localized Marketing Surface

Accuracy and consistency of the public-facing claims a prospect reads before installing, across languages.

Mapped capabilities

4 capabilities

Adoption claim consistency
195K GitHub stars, 950 contributors, 13,000+ commits, and 16M monthly developers as stated figures.
FAQ coverage
the documented FAQ set: what OpenCode is, how to use it, subscription requirements, reusing existing subscriptions, non-terminal use, cost, data and privacy, and open source status.
Localization parity
en, zh, ko, de, and es pages carrying the same feature list, figures, and privacy claims.
Open source positioning
open source identity, the Anomaly brand, and links to brand, privacy, and terms.

Coverage is mapped from OpenCode's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for OpenCode test?+

The coverage map is generated from OpenCode's own public product surface (open source AI coding agent): 6 scoring areas — Install & Platform Setup, Model & Provider Connectivity, and Agent Session Workflow, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the OpenCode evals scored?+

Every case generated for OpenCode — across Install & Platform Setup and Model & Provider Connectivity and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the OpenCode library include?+

The full OpenCode library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Package manager install paths and Platform guidance under Install & Platform Setup); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against OpenCode or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped OpenCode areas and set them up in a Corsac workspace, where you can run every test case against OpenCode or your own agent with your own data.