Eval directory

Evals for Capy

Eval coverage for Capy, mapped from its public product surface.

About Capy

Capy is a developer tools product that runs coding agents in the cloud. The provided pages contain only the site title, static assets, and a web app manifest, so no further capability detail is verifiable. The manifest classifies it under developer tools and productivity and describes it as an installable, standalone web app.

Industry

cloud developer tools / coding agent runtime

Website

capy.ai

Use the eval library for Capy

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Capy?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Cloud Agent Execution (headline claim)

The single verifiable product claim: 'Capy runs your coding agents in the cloud.' No page copy, API, or docs in the provided evidence describe how runs are started, configured, or observed, so coverage here is scoped to validating the claim surface itself rather than execution mechanics.

“Capy runs your coding agents in the cloud.” capy.ai

Mapped capabilities

4 capabilities

Claim consistency across surfaces
Manifest description, title, and any rendered marketing copy state the same cloud-agent value proposition.
Product categorization fit
Behavior matches the manifest's 'developer tools' and 'productivity' categories rather than an unrelated domain.
Entry point to a run surface
From start_url, an authenticated user reaches a surface where agent work can be initiated; existence only, not run semantics.
Unverified capability boundary
Flags any agent capability asserted in a pitch that is not evidenced by the live product — routed to verification, not scored.

Installable App Shell & Manifest

Capy ships a web app manifest at /api/manifest declaring an installable, standalone PWA with scope '/', start_url '/', theme and background color #EFEFF0, and 192/512/512-maskable icons. This is the packaging contract users hit on install.

Mapped capabilities

4 capabilities

Manifest validity and completeness
Required fields (name, short_name, start_url, scope, display, icons) are present, well-formed, and served as JSON.
Standalone display behavior
Installed app launches without browser chrome and honors declared theme/background colors during splash and load.
Icon set and maskable rendering
192px, 512px, and maskable 512px icons resolve and render without distortion or transparent-edge clipping.
Scope and start_url containment
Navigation from start_url stays within the declared '/' scope; out-of-scope links hand off to the browser as expected.

Illustrative example

Input: Fetch https://capy.ai/api/manifest and each icon URL it references, then install the app from a Chromium browser and launch it.
Expected behavior: The manifest parses as JSON with name, short_name, start_url, scope, display, and icons present. All three declared icons resolve successfully, and the installed app opens at start_url in a standalone window without browser chrome.

Authentication & Session Access

A Clerk client chunk is bundled and imported by routing code, indicating sign-in, session, and protected-route handling are part of the app shell. Coverage targets access control at the boundary, not identity-provider internals.

Mapped capabilities

4 capabilities

Sign-in and sign-out flow
User can authenticate and terminate a session; post-sign-out state exposes no authenticated view.
Protected route gating
Unauthenticated requests to in-app routes resolve to an auth surface rather than leaking app data.
Session persistence in installed app
Session survives relaunch of the standalone/installed window and normal page reloads.
Deep-link return after auth
A gated deep link, once authenticated, resolves to the originally requested route rather than a generic landing view.

Illustrative example

Input: In a browser profile with no Capy session, request an in-app deep link under the app scope rather than the root start_url.
Expected behavior: The request resolves to the app shell and lands on the sign-in surface instead of rendering authenticated content. No account, workspace, or agent-run data appears in the served response or the rendered view.

Client-Side Routing & Deep Links

The link chunk implements path segment parsing with required ($param), optional, wildcard ($), and prefix/suffix patterns plus case sensitivity and an LRU parse cache — a client-routed SPA where URLs are the addressable surface.

Mapped capabilities

4 capabilities

Dynamic and wildcard segment resolution
Required, optional, and catch-all segments resolve to the correct view for well-formed URLs.
Direct-load deep links
Pasting or reloading an in-app URL renders the same view as in-app navigation, not a hard 404.
History and back/forward behavior
Browser and in-app-window navigation history stays consistent with the rendered route.
Malformed path handling
Unparseable or unknown paths produce a defined not-found state instead of a blank or broken shell.

Asset Delivery & Load-Failure Recovery

The preload helper injects modulepreload/stylesheet links, propagates a csp-nonce meta value, tolerates already-present links, and dispatches a cancelable 'vite:preloadError' event before rethrowing — an explicit, testable degradation path.

Mapped capabilities

4 capabilities

Chunk load failure signaling
A failed module or CSS preload dispatches vite:preloadError and surfaces an error rather than hanging silently.
Recovery after transient failure
Retry or reload after a blocked chunk restores a working app shell without manual cache clearing.
CSP nonce propagation
Injected preload and stylesheet links carry the page's csp-nonce so assets load under a strict policy.
Duplicate preload suppression
Repeat navigations do not re-inject links for already-loaded modules or stylesheets.

UI Interaction Layer & Accessibility

Menu, popover, focus, and open-interaction chunks (floating-UI style) plus icon components that set role='img', aria-label, and aria-hidden indicate a keyboard- and screen-reader-relevant interaction layer worth validating independently of product depth.

Mapped capabilities

4 capabilities

Menu and popover keyboard operation
Open, traverse, submenu entry, and dismiss all work via keyboard alone.
Focus management and restore
Focus moves into an opened overlay and returns to the trigger on close.
Icon and control labeling
Decorative icons are aria-hidden; meaningful ones expose an accessible name via role and label.
Overlay positioning and resize
Popovers stay within the viewport and reposition correctly on window or content resize.

Coverage is mapped from Capy's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Capy test?+

The coverage map is generated from Capy's own public product surface (cloud developer tools / coding agent runtime): 6 scoring areas — Cloud Agent Execution (headline claim), Installable App Shell & Manifest, and Authentication & Session Access, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Capy evals scored?+

Every case generated for Capy — across Cloud Agent Execution (headline claim) and Installable App Shell & Manifest and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Capy library include?+

The full Capy library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Claim consistency across surfaces and Product categorization fit under Cloud Agent Execution (headline claim)); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Capy or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Capy areas and set them up in a Corsac workspace, where you can run every test case against Capy or your own agent with your own data.