Eval directory

Evals for Steel

Eval coverage for Steel, mapped from its public product surface.

About Steel

Steel is an open-source browser API that lets developers control fleets of cloud browsers for AI agents. It gives agents a real browser that can navigate pages, fill forms, solve CAPTCHAs, persist authentication, and extract content, with SDKs across multiple languages. It can be used as a hosted service with metered pricing or self-hosted via Docker or Railway.

Industry

cloud browser infrastructure for AI agents

Website

steel.dev

Use the eval library for Steel

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Steel?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Session Lifecycle & Fleet Control

Creating, observing, and tearing down cloud browser sessions at fleet scale through the Sessions API, including the launch-to-ready path Steel benchmarks publicly.

“Steel is an open source browser API that lets you control fleets of browsers in the cloud.” steel.dev

Mapped capabilities

4 capabilities

Session create and terminate
Opening and closing a cloud browser session via the Sessions API and CLI, including the browser lifecycle and control-plane behavior Steel publishes benchmarks for.
Fleet-level concurrency
Driving multiple cloud browsers at once rather than a single session, as described by Steel's 'control fleets of browsers in the cloud' surface.
Live session viewing
Watching an in-flight session while an agent works, as shipped in the Hermes Agent plugin and Stripe Projects integrations.
Session state teardown
What survives and what is discarded when a session ends, and how that interacts with persisted profiles.

Agent Web Actions

What an agent can actually do inside a Steel browser on a modern website: navigate, interact, clear interstitials, and pull content back out.

“navigate pages, fill forms, solve CAPTCHAs, persist auth, and extract content from modern websites” steel.dev

Mapped capabilities

4 capabilities

Page navigation
Directing the browser to URLs and moving through multi-step flows such as the flight-search example on Steel's homepage.
Form filling
Entering values and submitting forms on real sites as part of an agent task.
CAPTCHA solving
Clearing CAPTCHA challenges mid-task, including the entitlement path that unlocks solving.
Content extraction
Pulling structured or readable content off modern pages, including the `steel scrape` verification path.

Persistent Identity & Browser State

Making an agent look like the same returning visitor across runs, spanning auth persistence, profiles, leased IPs, and the agent-oriented Chromium fork.

“Custom Chromium fork built for agents that run the web, not just click through it.” steel.dev

Mapped capabilities

4 capabilities

Auth and cookie persistence
Login sessions, cookies, and credentials surviving across separate runs.
Dedicated IPs on a profile
Leasing a stable IP and attaching it to a profile so an agent returns as the same signed-in visitor every run.
Extension persistence
Browser extensions carried across runs as part of persistent state.
Stealth Browser runtime
The custom Chromium fork built for agents, as shipped on Launch Week v3 Day 1.

Illustrative example

Input: My agent has to log into a vendor portal every single run and keeps getting flagged as new. How do I make it come back as the same visitor?
Expected behavior: Points to persisting auth state in a profile and leasing a dedicated IP attached to that profile so the agent returns as the same signed-in visitor. Does not invent capabilities Steel has not shipped.

Network Reliability & Proxy Controls

Proxy configuration and its measured effect on run reliability — the area Steel's own blog identifies as a leading source of confusing failures.

Mapped capabilities

3 capabilities

Managed vs customer-supplied proxies
Choosing between Steel-managed proxies and bring-your-own, and the reliability gap Steel measured between them.
Per-site proxy and CAPTCHA controls
Setting proxy and CAPTCHA behavior per destination site rather than globally, as exposed in the Hermes Agent plugin.
Failure attribution
Telling the user which layer failed when a run breaks, instead of surfacing an undifferentiated error.

SDKs, CLI & Framework Integration

Getting Steel into a team's existing stack: language SDKs, the coding-agent CLI and skill, and framework-agnostic compatibility with existing automation scripts.

“Steel is the fastest remote browser in independent benchmarks: 0.89s average lifecycle, 1.7x to 9x faster than alternatives.” steel.dev

Mapped capabilities

4 capabilities

Multi-language SDKs
Driving Steel from the SDK surfaces Steel ships, including the Rust and Go SDKs added in Launch Week v3.
CLI and steel-browser skill setup
Install, authenticate, and verify flow documented at setup.steel.dev for coding agents.
Playwright/Puppeteer/Selenium compatibility
Moving existing local automation scripts onto Steel's cloud browsers under its framework-agnostic claim.
Host application integrations
Steel embedded in another product's agent surface, such as the Hermes Agent plugin and Stripe Projects.

Deployment, Pricing & Usage Visibility

How teams run and pay for Steel: hosted metered service versus self-hosting, and whether costs are legible before and after an agent runs.

“Run Steel in your own infrastructure with Docker or Railway one-click deploy. No vendor lock-in.” steel.dev

Mapped capabilities

4 capabilities

Self-hosting via Docker or Railway
Running Steel in the team's own infrastructure, including the Railway one-click deploy path.
Metered plans and rates
The three-plan, one-metering-model structure and rates a builder can read before an agent runs.
Credits and CAPTCHA unlock
The Launch credit grant and the deposit that unlocks CAPTCHA solving without a monthly subscription.
Usage dashboard
Post-run visibility into where spend went, per the pricing update.

Illustrative example

Input: Can I test whether Steel clears the CAPTCHA on my target site before I commit to a monthly plan?
Expected behavior: Confirms yes: Launch provides $30 in credits with no subscription, and CAPTCHA solving is unlocked with a $10 deposit that the credits then cover, so the workflow can be proven first.

Coverage is mapped from Steel's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Steel test?+

The coverage map is generated from Steel's own public product surface (cloud browser infrastructure for AI agents): 6 scoring areas — Session Lifecycle & Fleet Control, Agent Web Actions, and Persistent Identity & Browser State, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Steel evals scored?+

Every case generated for Steel — across Session Lifecycle & Fleet Control and Agent Web Actions and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Steel library include?+

The full Steel library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, Session create and terminate and Fleet-level concurrency under Session Lifecycle & Fleet Control); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Steel or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Steel areas and set them up in a Corsac workspace, where you can run every test case against Steel or your own agent with your own data.