Eval directory

Evals for Superserve

Eval coverage for Superserve, mapped from its public product surface.

About Superserve

Superserve provides durable Firecracker microVM sandboxes designed as persistent computers for long-horizon AI agents. Sandboxes can be snapshotted, forked, paused indefinitely, and resumed with the same state, with controlled network egress and credential access. It works with third-party harnesses (Claude Code, Codex, OpenClaw, Hermes and others), is billed per second, and offers a self-hostable open-source option plus an Anthropic integration where Claude Managed Agents run tool calls inside Superserve sandboxes.

Industry

agent sandbox / VM infrastructure

Headquarters

San Francisco

Website

www.superserve.ai

Use the eval library for Superserve

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Superserve?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Sandbox lifecycle and durability

Creating, running, pausing indefinitely, and resuming microVM sandboxes with state intact, including behavior across long idle stretches where no 24-hour session limit applies.

“Durable Firecracker VMs that pause indefinitely, resume instantly, and run as long as your agents need.” www.superserve.ai

Mapped capabilities

4 capabilities

Create and run a sandbox
Provisioning a named sandbox and executing commands inside it via the SDK.
Pause indefinitely
Suspending a sandbox with no session time limit and the resource accounting that applies while paused.
Resume with identical state
Restoring filesystem, installed packages, and working state after an arbitrary idle period.
Long-horizon unattended runs
Sessions that persist across multi-hour or overnight agent work such as multi-file refactors and pipelines.

Illustrative example

Input: Create a sandbox, install a global npm package and write a file under the working directory, then pause the sandbox. Days later, resume it and inspect the environment.
Expected behavior: The resumed sandbox is the same machine, not a fresh one. The installed package and the written file are still present with unchanged contents, and commands run without reprovisioning. No session time limit terminates the paused sandbox.

Snapshot, fork, and parallel exploration

Branching a workspace into isolated variants and returning to earlier points, supporting multi-agent systems that explore alternatives and select a best result.

Mapped capabilities

4 capabilities

Snapshot at an arbitrary step
Capturing VM state mid-task as a restorable point.
Fork a workspace
Deriving multiple independent sandboxes from one snapshot.
Parallel isolated agents
Running concurrent agents in separate sandboxes without cross-contamination.
Resume from a prior state
Rolling back to an earlier snapshot after a branch is discarded.

Harness and SDK integration

Bring-your-own-harness support across Claude Code, Codex, opencode, kilo code, Hermes, and OpenClaw, plus the TypeScript/Python SDK and the Anthropic Claude Managed Agents integration.

“Anthropic runs the agent loop, and tools execute in Superserve sandboxes.” www.superserve.ai

Mapped capabilities

4 capabilities

Bring your own harness
Installing and driving a third-party agent CLI inside the sandbox.
SDK surface (TS and Python)
Sandbox.create, commands.run, pause, and resume across both documented SDK languages.
Agent-inside vs. tools-inside topology
Running the full agent in the sandbox versus keeping the harness outside and sandboxing only tool calls.
Claude Managed Agents integration
Anthropic-run agent loop with bash/read/write tool calls executing in a Superserve microVM and results returned to the model.

Isolation, egress, and credential handling

Secure-by-default controls: microVM isolation, controlled network egress, and Superserve Secrets, which substitutes a stand-in token for the real key at the network edge only for allowed hosts.

“Secure by default Isolation · Network · Credentials” www.superserve.ai

Mapped capabilities

4 capabilities

Controlled network egress
Restricting which destinations sandboxed code may reach.
Stand-in token substitution
Edge-side swap of a placeholder for the real credential so keys never enter the sandbox.
Allowed-host scoping
Applying substitution only to permitted hosts and not to others.
Sandbox isolation boundary
Containment of AI-generated code, browsers, and system tasks within the microVM.

Illustrative example

Input: Configure a secret for an allowed API host, then from inside the sandbox make one request to that allowed host and one to a non-allowed host, both sending the stand-in token.
Expected behavior: The allowed-host request arrives at the destination carrying the real credential, substituted at the network edge. The non-allowed-host request is not given the real credential. At no point is the real key readable from inside the sandbox.

Data and repository connectivity

Connecting sandboxes to customer data and code sources shown in the platform architecture: S3, GCS, R2, GitHub, and GitLab.

“Paused sandboxes are billed only for storage.” www.superserve.ai

Mapped capabilities

3 capabilities

Object storage sources
Attaching S3, GCS, or R2 data to a sandbox workspace.
Git provider access
Working against GitHub or GitLab repositories from inside a sandbox.
Workspace persistence of fetched data
Retention of connected data across pause and resume.

Deployment model, billing, and recipes

How teams adopt the platform: managed pay-as-you-go with per-second billing, the self-hostable open-source option, enterprise on-premise, and the published starter recipes.

“Open source Self-hostable” www.superserve.ai

Mapped capabilities

4 capabilities

Per-second usage billing
vCPU, memory, and storage rates, including storage-only charges for paused sandboxes.
Free start and credits
No-credit-card start and the startup credits program.
Self-hosted open source
Running the platform yourself rather than on the managed service.
Starter recipes
Persistent dev environments, parallel sandboxed benchmarks, and pause-and-resume research agents.

Coverage is mapped from Superserve's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Superserve test?+

The coverage map is generated from Superserve's own public product surface (agent sandbox / VM infrastructure): 6 scoring areas — Sandbox lifecycle and durability, Snapshot, fork, and parallel exploration, and Harness and SDK integration, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Superserve evals scored?+

Every case generated for Superserve — across Sandbox lifecycle and durability and Snapshot, fork, and parallel exploration and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Superserve library include?+

The full Superserve library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, Create and run a sandbox and Pause indefinitely under Sandbox lifecycle and durability); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Superserve or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Superserve areas and set them up in a Corsac workspace, where you can run every test case against Superserve or your own agent with your own data.