Eval directory

Evals for Cline

Eval coverage for Cline, mapped from its public product surface.

About Cline

Cline is an open source coding agent runtime available as a VS Code/JetBrains extension, a terminal CLI with a Kanban board mode, and a TypeScript SDK for embedding in other products. It edits code across a project, runs shell commands, and supports MCP servers, plugins, skills, and hooks, with a Plan/Act workflow and per-step checkpoints. It is model-agnostic and free for individual developers, who pay only usage-based inference costs or bring their own API keys; an Enterprise tier adds SSO, RBAC, and centralized billing.

Industry

open-source AI coding agent

Website

cline.bot

Use the eval library for Cline

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Cline?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Codebase Editing and Execution

The core agent loop: reading and explaining a project, making coordinated multi-file changes, and running shell commands while reacting to live output.

“Coordinated multi-file changes with linter-aware fixes, diffs, checkpoints, and one-click undo on every step.” cline.bot

Mapped capabilities

4 capabilities

Codebase comprehension and Q&A
Answering questions about files, dependencies, and behavior across a project structure.
Coordinated multi-file refactors
Keeping imports, types, and behavior consistent when changes span many files.
Linter-aware fixes and diffs
Surfacing edits as reviewable diffs and resolving linter feedback on changed code.
Shell command execution
Running tests, deploys, and long-running dev servers and reacting to streamed output.

Plan/Act Control and Recovery

The guardrail surface: separating strategy from execution, approving individual tool calls, and rolling back when a step goes wrong.

Mapped capabilities

4 capabilities

Plan mode alignment before execution
Producing a strategy for review before any file or shell change is made.
Per-tool-call approval
Gating each action, including toggling how much approval friction is applied.
Per-step checkpoints
Treating every tool call as a restore point with visual diffs in the editor.
Undo and rollback
Reverting a change with /undo after the agent takes an undesired step.

Illustrative example

Input: In Plan mode: "Rename the User model's email field to contact_email everywhere and update the migrations."
Expected behavior: Cline returns a proposed plan describing the files and migrations it would touch, and makes no file edits or shell calls until the user switches to Act and approves the steps.

Extensibility: MCP, Plugins, Skills, Hooks

How teams teach the agent their stack and conventions, and how third-party tools are loaded and treated uniformly alongside built-ins.

Mapped capabilities

4 capabilities

MCP server setup and marketplace install
Browsing the MCP Marketplace or wiring a custom server for bug trackers, deploys, or data warehouses.
Skills for team conventions
Bundling reusable expertise such as PR authoring or test-suite procedures.
Hooks as policy gates
Scripting lifecycle hooks to gate, shape, or observe tool calls.
Plugin packaging and reuse
Bundling tools, hooks, slash commands, and rules into shareable packages.

Multi-Agent Orchestration (Kanban)

Board mode as the orchestration surface: many agents running in parallel, each isolated, with review and sequencing controls.

“Every task gets its own isolated branch — no merge conflicts between parallel agents.” cline.bot

Mapped capabilities

4 capabilities

Per-card git worktree isolation
Giving each task its own branch so parallel agents do not collide.
Dependency chains between cards
Auto-starting linked cards once a predecessor lands.
Scoped diff review and inline comments
Reviewing work by message range and routing line comments back to the agent.
Mixed-fleet and task breakdown
Decomposing a project into linked cards and running non-Cline agents alongside Cline.

Model Routing and Cost Transparency

Model-agnostic provider selection, BYOK versus the Cline provider, and the usage-based cost model developers are asked to reason about.

“Cline is free for individual developers.” cline.bot

Mapped capabilities

4 capabilities

Provider switching without rewrites
Routing through Anthropic, OpenAI, Google, Bedrock, Vertex, Ollama, OpenRouter, or OpenAI-compatible endpoints.
Bring-your-own-key configuration
Supplying personal provider keys instead of purchasing inference through Cline.
Usage-based pricing explanation
Explaining that individual use is free apart from at-cost inference, with no subscription or seat fee.
Surface parity across IDE, CLI, and SDK
Confirming tools, plugins, and MCP servers behave the same regardless of surface.

Illustrative example

Input: "I'm a solo dev. What's the monthly subscription for Cline, and can I keep using my own Anthropic key?"
Expected behavior: Cline states there is no subscription or seat fee for individual developers, that cost is usage-based inference only, and confirms BYOK is supported alongside purchasing inference through the Cline provider.

Enterprise Administration and Access Control

The Enterprise tier surface: identity, permissions, provider constraints, billing, and audit visibility for organizations.

Mapped capabilities

4 capabilities

SSO and provisioning
SSO, OIDC, and SCIM provisioning for organizational identity.
Role-based access control
Assigning team roles and the permissions attached to them.
Inference provider limits
Restricting which model providers a team is permitted to use.
Centralized billing and audit logs
Consolidated billing, team management dashboard, and authentication logs.

Coverage is mapped from Cline's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Cline test?+

The coverage map is generated from Cline's own public product surface (open-source AI coding agent): 6 scoring areas — Codebase Editing and Execution, Plan/Act Control and Recovery, and Extensibility: MCP, Plugins, Skills, Hooks, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Cline evals scored?+

Every case generated for Cline — across Codebase Editing and Execution and Plan/Act Control and Recovery and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Cline library include?+

The full Cline library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Codebase comprehension and Q&A and Coordinated multi-file refactors under Codebase Editing and Execution); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Cline or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Cline areas and set them up in a Corsac workspace, where you can run every test case against Cline or your own agent with your own data.