Eval directory

Evals for Cosine

Eval coverage for Cosine, mapped from its public product surface.

About Cosine

Cosine is an AI coding agent built for production software engineering work, available across CLI, cloud, and desktop surfaces. It is powered by Lumen, a family of post-trained coding models — including Lumen Outpost — specialised for code maintainability, agent behaviour, and niche or legacy languages. Cosine also offers enterprise and private deployments, and is building Lumen Sovereign, a UK-built model for critical sectors.

Industry

AI coding agent and specialist coding model

Website

cosine.sh

Use the eval library for Cosine

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Cosine?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Code Quality & Anti-Slop

Cosine's central claim is that Lumen is optimised for maintainability, readability, and production-quality output rather than volume of generated code. This area covers whether changes fit the existing codebase instead of accreting duplicated logic and unnecessary abstraction.

“Optimised for maintainability, readability, and production-quality outputs, not bloated code generation.” cosine.sh

Mapped capabilities

4 capabilities

Reuses existing abstractions
Extends helpers, constants, and patterns already present rather than introducing parallel implementations.
Change scope discipline
Diffs stay confined to the requested change without opportunistic refactors or unrelated file edits.
Maintainability over cleverness
Prefers readable, conventional structure to bloated or over-abstracted patches.
Comment and documentation restraint
Matches the surrounding code's commenting density instead of narrating every line.

Illustrative example

Input: Add retry with exponential backoff to this Python HTTP client. It already has a `_request` helper and a module-level `RETRY_LIMIT` constant.
Expected behavior: The agent adds backoff inside the existing `_request` helper and reuses `RETRY_LIMIT`, rather than introducing a second retry wrapper, decorator, or duplicate constant. The diff stays inside the client module with no unrelated refactors.

Niche & Legacy Languages

Lumen Outpost is post-trained for languages where generalist models degrade, explicitly including COBOL, Fortran, Verilog, Rust, and complex SQL, and for long-lived or environment-constrained systems. This area covers competence and honesty outside mainstream stacks.

“Deploy models where your data lives, with security, visibility, and ownership built in.” cosine.sh

Mapped capabilities

4 capabilities

Legacy language edits
Correct, idiomatic changes in COBOL and Fortran codebases including fixed-format and copybook conventions.
Hardware and constrained environments
Verilog and other environment-constrained targets where toolchain assumptions matter.
Complex SQL reasoning
Multi-join, dialect-specific queries and migrations against existing schemas.
Reasoning across long-lived architectures
Locating and changing behaviour in large legacy codebases without breaking adjacent call sites.

Agent Behaviour & Interaction

Cosine trains directly against the interaction failures it names: verbosity, overconfidence, sycophancy, and declaring tasks complete before they are. This area covers the behavioural signals Vibe-Bench targets — initiative, planning, and update substance.

Mapped capabilities

4 capabilities

Honest completion reporting
Says what was and was not done, including failing tests, rather than claiming premature success.
Planning before large changes
Produces a plan proportional to the size of the change and follows it.
Update substance
Progress messages carry real information instead of filler acknowledgement.
Pushback without sycophancy
Flags a flawed premise once, then proceeds on the user's decision.

Surfaces & Execution Model

Cosine ships as a terminal-native CLI, a cloud workspace, and a desktop app. The documented mechanics — local-to-remote execution, MCP tool access, multi-agent orchestration, and parallel cloud tasks — define what a user can actually do on each surface.

“a local-to-remote execution model, external tool access via MCP, and multi-agent orchestration” cosine.sh

Mapped capabilities

4 capabilities

CLI local-to-remote execution
Terminal workflow where work moves between local and remote execution contexts.
External tools via MCP
Connecting and using external tool access through MCP from the CLI.
Multi-agent orchestration
Coordinating multiple agents on a single body of work.
Parallel cloud tasks and shared projects
Running tasks in parallel and collaborating with engineers, PMs, and stakeholders in one project.

Deployment, Security & Sovereignty

Cosine markets deploying models where the customer's data lives, with security, visibility, and ownership, plus enterprise and private deployments and the in-progress UK-built Lumen Sovereign model. This area covers accuracy and restraint when answering deployment questions.

“Enterprise and private deployment pricing is scoped with sales because infrastructure, support, and security requirements vary.” cosine.sh

Mapped capabilities

4 capabilities

Private and enterprise deployment options
Describes what enterprise and private deployment covers without inventing unpublished guarantees.
Air-gapped versus cloud tradeoffs
Represents the published cost and isolation comparison faithfully.
Lumen Sovereign status
Treats Sovereign as in development with a public request for input, not as generally available.
Data residency and ownership claims
Stays within stated security, visibility, and ownership positioning.

Plans, Credits & Billing

Pricing is credit-based across Starter, Team, and Enterprise tiers, with add-on credits and sales-scoped enterprise deployments. This area covers whether published numbers are quoted exactly and unpublished terms are not fabricated.

Mapped capabilities

4 capabilities

Tier pricing accuracy
Correct monthly price and included credits for Starter, Team, and Enterprise.
Add-on and top-up handling
Correct per-1M add-on rates and the fact that all tiers can buy top-ups.
What credits represent
Usage across agent work, model calls, and cloud execution, varying with task size, model, and runtime.
Custom and enterprise quoting
Routes custom deployment pricing to sales instead of quoting a number.

Illustrative example

Input: We're a team of eight shipping daily. What does the Team plan cost, how many credits do we get, and what happens if we run out mid-month?
Expected behavior: States $199/month with 47M credits/month, and that add-on credits are available at $5.00 per 1M, purchasable as top-ups at any time on all tiers. No invented per-seat charges, throttling behaviour, or overage penalties.

Coverage is mapped from Cosine's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Cosine test?+

The coverage map is generated from Cosine's own public product surface (AI coding agent and specialist coding model): 6 scoring areas — Code Quality & Anti-Slop, Niche & Legacy Languages, and Agent Behaviour & Interaction, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Cosine evals scored?+

Every case generated for Cosine — across Code Quality & Anti-Slop and Niche & Legacy Languages and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Cosine library include?+

The full Cosine library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Reuses existing abstractions and Change scope discipline under Code Quality & Anti-Slop); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Cosine or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Cosine areas and set them up in a Corsac workspace, where you can run every test case against Cosine or your own agent with your own data.