Eval directory

Evals for Skyvern

Eval coverage for Skyvern, mapped from its public product surface.

About Skyvern

Skyvern uses AI agents to automate workflows on any website directly in the browser, without APIs or custom scripts. It handles real-world obstacles like CAPTCHAs, 2FA/TOTP logins, and proxy/geo targeting, and extracts structured data in formats such as CSV or JSON. Industry-specific pages cover fintech and banking, HR tech and recruiting, insurance, and healthcare.

Industry

AI browser automation agents

Website

www.skyvern.com

Use the eval library for Skyvern

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Skyvern?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Browser Workflow Execution

Turning a described workflow into correct, completed actions in a live browser, on sites with no API and no custom script — the core claim of the product.

“Full audit trail for every workflow execution.” www.skyvern.com

Mapped capabilities

4 capabilities

Natural-language workflow interpretation
Mapping a plain-English task description onto the right sequence of on-page actions without additional configuration.
Form filling and submission
Locating fields, entering supplied values, and submitting forms including multi-page and conditional forms.
Portal navigation and document retrieval
Navigating to the correct section of an unfamiliar portal and downloading target documents such as invoices or statements.
Multi-step task completion
Carrying state across steps and finishing the whole workflow rather than stopping at a partial result.

Authenticated Access and 2FA

Getting into accounts that require real credentials and second factors, since most of the described work happens behind a login.

Mapped capabilities

4 capabilities

Standard portal login
Completing username and password flows and confirming the authenticated state was actually reached.
2FA and TOTP handling
Submitting the second-factor or TOTP code automatically as part of the workflow rather than handing off to a human.
Credential handling within a workflow
Using supplied secrets only for the intended login step and not exposing them in outputs or summaries.
Session continuity
Staying authenticated across the remaining steps of a workflow after login succeeds.

Illustrative example

Input: Log into the payroll portal using the stored credentials and continue to the dashboard. The account requires a TOTP code at the second authentication step.
Expected behavior: The agent submits the password, then supplies a current TOTP code at the second step and reaches the authenticated dashboard on its own. It does not pause to ask a person for the code, and the credentials never appear in the run output.

CAPTCHA and Challenge Handling

Keeping runs alive through the bot-challenge obstacles that stop conventional automation, which the site positions as solved natively without third-party services.

“Skyvern solves CAPTCHAs natively so your automations keep running without interruption” www.skyvern.com

Mapped capabilities

3 capabilities

Native CAPTCHA solving
Resolving CAPTCHA challenges in-product rather than routing to an external solving service.
Challenge detection mid-run
Recognizing that a challenge has interrupted the flow instead of proceeding against a blocked page.
Resumption after a challenge
Returning to the original task and completing the remaining steps once the challenge is cleared.

Proxy Networks and Geo Targeting

Controlling where a browser session appears to originate, at country, state, or zip-code granularity, across residential and datacenter networks.

“Support for country, state, or even precise zip-code level targeting across residential and datacenter proxy networks.” www.skyvern.com

Mapped capabilities

4 capabilities

Country-level routing
Honoring a requested country such as US, Canada, EU, India, Japan, or Brazil for the session.
State and zip-code targeting
Applying finer-grained regional targeting when the workflow specifies a state or precise zip code.
Residential versus datacenter selection
Using the requested proxy network type for the run.
Geo-conditional page handling
Working correctly with content that differs by the targeted region.

Structured Data Extraction

Producing machine-usable output in the caller's own schema, in formats such as CSV or JSON, so results can be piped into downstream systems.

“Extract data from workflows in any schema of your choice including CSV or JSON formats.” www.skyvern.com

Mapped capabilities

4 capabilities

Caller-defined schema conformance
Returning exactly the fields requested, with the requested names and types.
JSON output
Emitting valid, parseable JSON with no surrounding prose.
CSV output
Emitting well-formed CSV with a stable header row and correct escaping.
Extraction fidelity to the page
Values in the output matching what the page actually displayed, with no invented rows or fields.

Illustrative example

Input: Download the three most recent invoices from this vendor portal and return invoice_number, issue_date, and total_amount for each, as JSON matching the schema I supplied.
Expected behavior: The response is a JSON array of three invoice objects carrying only the three requested fields, with values matching what the portal displayed. No explanatory prose wraps the JSON and no extra keys are added.

Observability and Regulated-Industry Fit

The explainability, audit, and compliance surface that determines whether an agent's work on authenticated systems is reviewable and acceptable in regulated settings.

Mapped capabilities

4 capabilities

Run summaries with action rationale
Reporting each action taken and why, in a form a reviewer can inspect after the fact.
Audit trail completeness
Every workflow execution producing a full trace covering the actions actually performed.
Trace fidelity to the run
The summary reflecting what really happened, including steps that failed or were retried.
Regulated-vertical workflow context
Handling the fintech, HR tech, insurance, and healthcare workflows the site calls out, under its stated HIPAA and SOC2 posture.

Coverage is mapped from Skyvern's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Skyvern test?+

The coverage map is generated from Skyvern's own public product surface (AI browser automation agents): 6 scoring areas — Browser Workflow Execution, Authenticated Access and 2FA, and CAPTCHA and Challenge Handling, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Skyvern evals scored?+

Every case generated for Skyvern — across Browser Workflow Execution and Authenticated Access and 2FA and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Skyvern library include?+

The full Skyvern library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, Natural-language workflow interpretation and Form filling and submission under Browser Workflow Execution); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Skyvern or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Skyvern areas and set them up in a Corsac workspace, where you can run every test case against Skyvern or your own agent with your own data.