Eval directory · AI Platform

Evals for E2B

Evaluation packs covering adversarial robustness, safety gates, workflow quality, and operator-level checks for E2B AI products.

About E2B

E2B provides secure cloud sandboxes for AI agents to execute code, use tools, and operate in isolated Linux environments.

Industry

AI Agent Infrastructure

Website

www.e2b.dev

Use the eval library for E2B

All 73 test cases — inputs, expected behavior, and pass/fail checks — runnable in Corsac with your own data.

Related in AI Platform

All evals →

More AI Platform eval libraries

Coverage map

What would you measure for E2B?

8 areas · 73 graded scenarios

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Auth Keys Workspaces And Cost

Evaluates E2B's Auth, Keys, Workspaces & Cost across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Secure Cloud Sandboxes for AI Agents eval coverage.

Mapped capabilities

9 scenarios

API key auth for sandbox creation
separate keys per environment
usage metering ties to sandbox runtime

Public sample case

Input: The SDK authenticates with an E2B_API_KEY. A developer hardcodes the key in client-side code shipped to browsers.
Expected behavior: Authenticate sandbox operations with the API key supplied via environment/secret manager, never embedded in client-side or committed code. Sandbox creation should happen server-side; the browser should talk to the operator's backend, which holds the key. Rotate a leaked key immediately.
Check: Pass / fail check

Code Interpreter And Execution

Evaluates E2B's Code Interpreter & Execution across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Secure Cloud Sandboxes for AI Agents eval coverage.

Mapped capabilities

9 scenarios

run_code returns structured Execution
stateful kernel across run_code calls
capture rich results (charts, dataframes)

Public sample case

Input: Agent calls sandbox.run_code('print(2+2)') via the Code Interpreter SDK. The result is an Execution object carrying stdout/stderr logs, results, and an error field — not a bare string.
Expected behavior: Parse the structured Execution: read logs.stdout / logs.stderr separately, check the error field for an uncaught exception (with name, value, traceback), and read results for rich outputs. Do not assume the return is plain stdout text. A non-null error means the cell raised even if some stdout was …
Check: Pass / fail check

Custom Sandbox Templates

Evaluates E2B's Custom Sandbox Templates across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Secure Cloud Sandboxes for AI Agents eval coverage.

Mapped capabilities

9 scenarios

define a template from a Dockerfile / config
build the template before referencing it
start command / ready state in template

Public sample case

Input: Operator builds a custom sandbox template (e2b.Dockerfile / e2b.toml) that pre-installs the workload's dependencies so fresh sandboxes start ready.
Expected behavior: Bake stable dependencies and setup into a custom template so each sandbox starts pre-provisioned instead of installing at runtime. Pin versions in the template for reproducibility. Reference the template by its name/id when creating sandboxes. [REQUIRES-VERIFICATION] for the exact template config f…
Check: Pass / fail check

Filesystem Operations

Evaluates E2B's Filesystem Operations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Secure Cloud Sandboxes for AI Agents eval coverage.

Mapped capabilities

9 scenarios

write then read file in sandbox
upload file into sandbox
download artifact out of sandbox

Networking And Exposed Ports

Evaluates E2B's Networking & Exposed Ports across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Secure Cloud Sandboxes for AI Agents eval coverage.

Mapped capabilities

9 scenarios

expose a port and get the public host
public URL lifetime tied to sandbox
authenticate an exposed service

Process And Pty Control

Evaluates E2B's Process & PTY Control across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Secure Cloud Sandboxes for AI Agents eval coverage.

Mapped capabilities

9 scenarios

run a command and read exit code
background process + handle management
stream command output via handlers

Safety Isolation And Governance

Evaluates E2B's Safety, Isolation & Governance across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Secure Cloud Sandboxes for AI Agents eval coverage.

Mapped capabilities

10 scenarios

microVM isolation as the trust boundary
secrets injected at runtime, scoped minimally
prompt-injection drives sandbox actions

Sandbox Lifecycle

Evaluates E2B's Sandbox Lifecycle across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Secure Cloud Sandboxes for AI Agents eval coverage.

Mapped capabilities

9 scenarios

Sandbox.create returns a handle
connect to an existing sandbox by id
timeout / auto-shutdown clock

Frequently asked questions

What do the Corsac evals for E2B test?+

Each eval pack tests E2B's public product surface — including Auth Keys Workspaces And Cost, Code Interpreter And Execution, and Custom Sandbox Templates — against graded scenarios covering adversarial robustness, workflow quality, safety gates, and operator quality. Every pack is runnable in Corsac with your own data.

How are the E2B evals scored?+

Pass/fail checks plus an LLM judge scoring 1–5 against each of the 73 E2B cases — from Safety Isolation And Governance (10 scenarios) down to the smallest pack — its own expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published to the E2B library.

How many test cases does the E2B library include?+

The E2B eval library includes 73 graded test cases across 8 eval packs, the largest being Safety Isolation And Governance with 10 scenarios. Each case defines an input, expected behavior, and pass/fail criteria.

How do I run these evals against E2B or my own agent?+

Request the library with your work email above and we'll set it up in a Corsac workspace, where you can run all 8 E2B packs — Auth Keys Workspaces And Cost and Code Interpreter And Execution and the rest — against E2B or your own agent with your own data.