Eval directory

Evals for QA.tech

Eval coverage for QA.tech, mapped from its public product surface.

About QA.tech

QA.tech is an autonomous AI testing platform whose agents test applications by intent through the real UI instead of scripted selectors. It covers web apps, mobile (iOS/Android), API and email/SMS flows, and runs continuously from pull request checks through production. It integrates with GitHub for PR testing and exposes an MCP server so coding agents like Cursor and Claude Code can trigger and create tests.

Industry

AI QA and end-to-end test automation platform

Website

qa.tech

Use the eval library for QA.tech

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for QA.tech?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Intent-based test authoring

Turning a plain-language description of a user journey into a runnable test that an agent drives through the live UI, without scripts, recordings, or selectors.

“Agents see your UI the way a user does – no selectors, no scripts.” qa.tech

Mapped capabilities

4 capabilities

Natural-language flow to test case
Chat-shaped intent becomes a test an agent can execute against the live app.
Goal-driven execution
Agent works toward the stated outcome rather than a fixed step sequence.
Resilience to UI change
Tests reach their goal through redesigns, DOM shifts, and framework migrations.
Compounding app knowledge
Crawled flows and screens build a knowledge graph that later tests build on.

Cross-surface coverage

Running the same intent across the surfaces a real journey touches: web apps, mobile web, native iOS and Android, APIs, and email or SMS steps.

Mapped capabilities

4 capabilities

Web application flows
Ecommerce, SaaS, and internal web apps driven through the rendered UI.
Native mobile iOS and Android
One described flow adapted across native, React Native, and Flutter apps.
Journeys spanning UI and API
A single scenario covering backend calls alongside the interface steps.
Email and SMS steps
Out-of-band verification and notification steps inside an end-to-end flow.

Pull request and CI validation

Testing every pull request in its preview environment and reporting the result back into the review workflow before merge.

“Connect GitHub and QA.tech automatically picks up every PR and its preview deployment” qa.tech

Mapped capabilities

4 capabilities

GitHub PR pickup
Connecting GitHub and detecting each PR and its preview deployment.
Pass/fail reporting to the PR
Results posted to GitHub so reviewers see what changed and what broke.
Merge blocking on failure
A broken regression stops the merge rather than landing on staging.
Parallel and scheduled runs
Concurrent PR validation plus continuing runs against deployed environments.

Illustrative example

Input: Open a pull request whose preview deployment breaks the checkout flow, and let the connected GitHub integration pick up the PR automatically.
Expected behavior: The agents run regression and exploratory tests against that PR's preview environment, post a failing result to the pull request identifying the broken flow, and the merge is blocked rather than allowed through.

MCP agent control surface

The MCP server that lets coding agents such as Cursor and Claude Code trigger runs, create tests, and read results without leaving the editor.

“QA.tech tests your product from the outside – no repo access, no lengthy security review.” qa.tech

Mapped capabilities

4 capabilities

Run control tools
start_run, list_runs, and get_run for triggering and polling test execution.
Test creation tools
create_chat and create_test_case for authoring coverage from the editor.
Environment discovery
list_application_environments and get_environment for targeting the right environment.
Local tunnel access
create_remote_tunnel so agents can reach a locally running build.

Illustrative example

Input: From an MCP client, call start_run for an environment resolved via list_application_environments, then poll get_run until the run reaches a terminal state.
Expected behavior: start_run returns a run identifier tied to the requested environment, list_runs includes that run, and get_run reports its status with a final pass or fail outcome once execution completes.

Failure diagnostics and triage

What a run returns when something breaks, and how that failure reaches the team's existing tracking tools.

“Full regression in minutes, blocks the merge when something breaks.” qa.tech

Mapped capabilities

4 capabilities

Step-by-step trace and video
A replay and ordered trace of what the agent did on each step.
Screenshots, logs, and network activity
Per-step evidence for diagnosing a failure without reproducing it by hand.
Ticket filing on failure
Failures routed into Linear or Jira as tickets.
Coverage and execution reporting
Coverage reports and test execution performance reports for the suite.

Environments, access, and data governance

How the platform is scoped and administered: environment configuration, seats and parallelism by plan, retention, and the security posture stated for enterprise review.

Mapped capabilities

4 capabilities

Environment configuration
Multiple environments per application, with counts varying by plan.
Seats and parallel run limits
User counts and concurrent test runs bounded by Starter, Growth, and Enterprise tiers.
Test data retention
Retention windows of 30 or 90 days, or custom on Enterprise.
External-only testing posture
Testing from outside the product with no repo access and no customer data used for model training.

Coverage is mapped from QA.tech's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for QA.tech test?+

The coverage map is generated from QA.tech's own public product surface (AI QA and end-to-end test automation platform): 6 scoring areas — Intent-based test authoring, Cross-surface coverage, and Pull request and CI validation, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the QA.tech evals scored?+

Every case generated for QA.tech — across Intent-based test authoring and Cross-surface coverage and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the QA.tech library include?+

The full QA.tech library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Natural-language flow to test case and Goal-driven execution under Intent-based test authoring); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against QA.tech or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped QA.tech areas and set them up in a Corsac workspace, where you can run every test case against QA.tech or your own agent with your own data.