Eval directory

Evals for Ellipsis

Eval coverage for Ellipsis, mapped from its public product surface.

About Ellipsis

Ellipsis is a cloud platform that runs coding agents in isolated sandboxes, where each agent is defined as a YAML file committed to your repository and deployed by merging to the default branch. It offers two products: Cloud Agents for feature development and workflow automation, and Code Review, which comments on pull requests as commits land. Pricing is usage-based, metered by tokens, CPU, and memory, with scoped permissions, spend caps, and full session logs.

Industry

cloud platform for coding agents / AI code review

Website

www.ellipsis.dev

Use the eval library for Ellipsis

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Ellipsis?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Agents as Code: definition and deploy

An agent is a single YAML file in the repository declaring instructions, trigger, repositories, permissions, and budget. The version on the default branch is the live agent, so changes ship by merging a pull request rather than through a deploy step.

“Deploy agents for feature development, code review, and more.” www.ellipsis.dev

Mapped capabilities

4 capabilities

YAML agent definition fields
Trigger, model, prompt, repositories, permissions, and budget expressed as fields in one file.
Merge-to-deploy and revert
Default-branch version is live; behavior changes via PR, with author, date, diff, and revert.
Trigger types
Cron schedule, GitHub / Linear / Sentry / Slack events, @ellipsis mention, or no trigger for on-demand runs.
Invalid config handling
An invalid edit records a sync error while the last valid configuration stays live.

Illustrative example

Input: I updated the prompt in agents/reviewer.yaml on my feature branch and pushed it. Is the new prompt live for my team's pull requests yet, or do I need to deploy it?
Expected behavior: States the change is not live because only the default-branch version of the file is the running agent, and that it takes effect by merging the pull request. Notes there is no separate deploy step.

Sandboxed agent execution

Each agent invocation runs in a short-lived, isolated sandbox with a cloned copy of the repository, restricted network access, and a time and resource budget, then shuts down without persisting state.

“It cannot touch production infrastructure, access secrets outside its scope, or persist state between invocations.” www.ellipsis.dev

Mapped capabilities

4 capabilities

Per-session isolation
Fresh container per invocation; agent can read and modify the clone but not production infrastructure.
Parallel sessions
Isolation lets concurrent agent sessions run without blocking one another.
Session lifecycle
Event wakes the agent, work is done, an artifact is delivered, the sandbox shuts down.
Delivered artifacts
Pull request, an answer in the thread that asked, or a report of merged work.

Code review on pull requests

The Code Review product reviews pull requests as commits land and posts findings as inline comments, enabled by one org-wide toggle with no YAML required.

“Ellipsis runs coding agents in cloud sandboxes and reviews every pull request.” www.ellipsis.dev

Mapped capabilities

4 capabilities

Org-wide enablement
Single toggle turns review on across the organization without an agent file.
Inline comments as commits land
Findings posted on the diff while the pull request continues to receive commits.
Optional pipeline file
Scopes which pull requests get reviewed or replaces the default reviewers.
Grounding in team context
Review checks the diff against the team's conventions and past review feedback.

Permissions and budget governance

Permissions and spend limits are explicit fields in the agent file, reviewed like any other code, so blast radius is visible before an agent runs and enforced while it runs.

“Permissions are explicitly defined in YAML, so there are no rogue agents.” www.ellipsis.dev

Mapped capabilities

4 capabilities

Scoped permissions in YAML
Explicitly declared access — described as less permission than a human — defined before the agent runs.
Spend caps at multiple scopes
Limits set per session, per developer, per day, or per month.
Cap-reached behavior
A session that exhausts its budget stops and is surfaced as parked at its spend cap.
Governance through review
Permission and budget changes arrive as pull requests with an author and a diff.

Illustrative example

Input: My dependency-bump agent shows as parked at its spend cap. Did it finish the work, and will it pick back up on its own once the day rolls over?
Expected behavior: Explains the session stopped when it hit the configured spend limit rather than completing the work, and that caps are enforced limits set in the agent file at session, developer, day, or month scope. Does not promise automatic resumption.

Observability and session control

Every step of a session is logged, including thinking and tool calls, and work can be traced back to the agent that produced it. Sessions are started, streamed, and inspected from the dashboard, the agent CLI, or the REST API.

Mapped capabilities

4 capabilities

Full step-level logs
Session transcripts covering agent thinking and tool calls.
Provenance of artifacts
Trace a pull request back to the agent and session that spawned it.
Control surfaces
Dashboard at app.ellipsis.dev, agent CLI, and REST API for starting and streaming sessions.
Session history
Reading past sessions and their outcomes after the sandbox has shut down.

Metering, billing, and model routing

Pricing is usage-based with no per-seat or idle charges: each session meters tokens, CPU, and memory plus a platform fee, with deployment options that change where token spend lands.

“Set spend limits per session, per developer, per day, or per month.” www.ellipsis.dev

Mapped capabilities

4 capabilities

Per-session metering
Tokens, CPU-hours, and memory-hours metered per session, with a platform fee on token usage.
Deployment and BYOK options
Ellipsis Cloud, BYOK Bedrock routing tokens to your AWS bill, or private cloud in your own account.
Session cost profiles
Code review meters tokens only with no sandbox; agent sessions also meter CPU and memory.
Model selection
Agents declare a model and work with the agents teams already use — Claude, Codex, Gemini.

Coverage is mapped from Ellipsis's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Ellipsis test?+

The coverage map is generated from Ellipsis's own public product surface (cloud platform for coding agents / AI code review): 6 scoring areas — Agents as Code: definition and deploy, Sandboxed agent execution, and Code review on pull requests, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Ellipsis evals scored?+

Every case generated for Ellipsis — across Agents as Code: definition and deploy and Sandboxed agent execution and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Ellipsis library include?+

The full Ellipsis library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, YAML agent definition fields and Merge-to-deploy and revert under Agents as Code: definition and deploy); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Ellipsis or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Ellipsis areas and set them up in a Corsac workspace, where you can run every test case against Ellipsis or your own agent with your own data.