Eval directory

Evals for QA Wolf

Mapped eval coverage for QA Wolf — adversarial robustness, safety gates, workflow quality, and operator-level checks across its public product surface.

Use the eval library for QA Wolf

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for QA Wolf?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Workflow Discovery & Test Mapping

Autonomous exploration of an application to produce a documented map of its workflows, including how the system solicits and incorporates human domain knowledge where exploration alone is insufficient, and how manual test plans, past P0s, and product specs feed coverage decisions.

“100% of teams achieve 80%+ automated test coverage in weeks.” www.qawolf.com

Mapped capabilities

4 capabilities

Autonomous app exploration
Crawls the product and documents discoverable workflows without a pre-written test plan.
Domain-knowledge gap filling
Identifies what exploration cannot infer and asks the team targeted questions to close the gap.
Coverage-gap and redundancy analysis
Reviews existing manual test plans to surface untested edge cases and duplicate coverage.
Grounding in specs and incident history
Uses roadmaps, specs/ACs, and past incident reports to test intended behavior, not just shipped behavior.

Illustrative example

During autonomous exploration of a B2B invoicing app, the agent reaches a 'Submit for approval' action that requires a second account with an approver role, which it has not been given. Produce the workflow map for the invoicing area. → The map should represent the approval workflow as discovered-but-blocked rather than dropping it or inventing the downstream steps. The agent should state what it could not reach, name the specific missing input (approver-role credentials), and surface that as a domain-knowledge question to the team — consistent with the documented behavior of getting domain knowledge from the user to fill gaps.

Test Authoring & Automation Breadth

Generation of production-grade test code across web (Chrome, Firefox, WebKit), iOS, Android, and Electron, including complex cases that resist conventional automation, and the setup/teardown discipline that makes generated tests safe to run repeatedly.

“One platform to manage the entire testing lifecycle: Map your workflows, automate web and mobile tests” www.qawolf.com

Mapped capabilities

4 capabilities

Web test generation across browsers
Produces Playwright-based tests targeting Chrome, Firefox, and WebKit.
Mobile and desktop coverage
Automates iOS and Android cases on emulators and real devices, plus Electron apps.
Hard-to-automate device capabilities
Handles Canvas API, barcode scanning, iBeacon/geo-fence, camera, and microphone cases.
Data setup and cleanup per test
Every generated test provisions and tears down its own data.

Parallel Execution & Run Orchestration

Containerized, fully parallel test execution and the coordination model that keeps concurrent runs from interfering, including how runs are triggered and how multi-actor scenarios are sequenced across devices, services, and time.

Mapped capabilities

4 capabilities

Per-test containerized isolation
Each test runs in its own container so unlimited parallel runs don't collide.
Trigger surfaces
Runs start manually, on a schedule, or on deploy via webhook.
Multi-user, multi-device orchestration
Coordinates tests where several actors or devices must interact in a defined order.
Collision and race-condition safeties
Built-in guards prevent shared-state interference under parallelism.

Failure Investigation & Release Signal

What happens after a test fails: reproduction by humans, separation of real defects from flakes, and the evidence package delivered to the customer team. This is the surface the zero-flake and human-verified-bug-report guarantees rest on.

“Guaranteed zero flakes. You will never be alerted to a test flake, only real bugs get flagged.” www.qawolf.com

Mapped capabilities

4 capabilities

Failure triage and reproduction
Every failure is investigated and reproduced before anything reaches the customer.
Flake suppression
Flakes are absorbed internally; the team is never alerted to one.
Bug report evidence package
Reports include video with reproduction steps, Playwright traces, and console logs.
Noise-free release signal
Only verified bugs are flagged, so suite status is directly actionable at release time.

Illustrative example

Two tests fail in the same parallel run. Test A: checkout submit returns HTTP 500, reproduced on 3 of 3 reruns. Test B: a dashboard assertion times out once, passes on 5 of 5 reruns with no application error. Decide what, if anything, is reported to the customer team and in what form. → Test A should be reported as a human-verified bug with the documented evidence package. Test B should be handled internally as a flake and must not generate a customer-facing alert, per the zero-flake guarantee; the correct disposition is internal investigation/repair, not notification.

Suite Maintenance & Coverage Growth

Keeping an existing suite healthy as the product changes, and the trajectory from zero coverage to the marketed 80%+ threshold, including the response-time commitments and consulting support that accompany the managed tier.

“We investigate every test failure for you and repair the test if needed.” www.qawolf.com

Mapped capabilities

4 capabilities

Test repair after UI changes
Updates tests when the application's interface shifts, rather than leaving them broken.
24-hour investigation and maintenance
Failures are investigated and tests repaired within a stated window by staff across three continents.
Coverage ramp to threshold
Progress toward the 80%+ automated coverage level claimed for teams in weeks to four months.
Code and infrastructure consulting
Advises on app-side changes that reduce flakiness at the source.

Commercial Model, Integration & Portability

How the two purchase paths are represented and how the product connects to and disconnects from a customer's stack: consumption pricing on the self-serve platform, tests-under-management pricing for the service tier, CI wiring, and the open-source export path that backs the no-lock-in claim.

“Export the open-source Playwright at any time.” www.qawolf.com

Mapped capabilities

4 capabilities

Consumption pricing transparency
AI credits at 1¢ and runner minutes at 15¢, and what each unit covers.
Platform vs. Coverage-as-a-Service fit
Distinguishes self-serve ownership from the managed tier priced per tests under management.
CI integration
Connects via API or webhook, including PR-branch smoke suites pre-merge.
Playwright export / no vendor lock-in
Customers can export the open-source Playwright code at any time.

Coverage is mapped from QA Wolf's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for QA Wolf test?+

The coverage map above is generated from QA Wolf's public product surface: 6 scoring areas spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the QA Wolf evals scored?+

Every eval set is graded the same way: pass/fail checks plus an LLM judge scoring 1–5 against each case's expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the QA Wolf library include?+

The full QA Wolf library is built on request. The coverage map spans 6 areas and 24 capabilities; each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against QA Wolf or my own agent?+

Request the library with your work email above. We'll build it out and set it up in a Corsac workspace, where you can run every test case against QA Wolf or your own agent with your own data.