Eval directory

Evals for Momentic

Mapped eval coverage for Momentic — adversarial robustness, safety gates, workflow quality, and operator-level checks across its public product surface.

Use the eval library for Momentic

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Momentic?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Test authoring and structure

Turning natural-language intent into readable, repeatable YAML tests that live in the customer's codebase — the format, its reusable units, and the variability the format has to absorb. Grounded in the docs' test-structure, best-practices, web/mobile steps references, guides on AI actions, test data and variants, and the July 2026 changelog additions to step syntax.

“Write end-to-end tests for web and mobile in plain English.” momentic.ai

Mapped capabilities

4 capabilities

YAML test format, modules, and variables
Test file anatomy; reuse via modules; variable interpolation such as {{ env.PASSWORD }}; readability as a review artifact in the repo.
Step vocabulary and targets
Web and mobile step references; selector targets accepting both CSS and auto-detected XPath; the fill alias versus per-keystroke type.
Declarative AI actions and phases
Rolling imperative steps up into declarative goals, grouping into semantic phases, and designing around documented agent limits.
Test data and variants
Inline fake-data generation and running one test across CSV input rows.

Agentic authoring and maintenance loop

The behavior of the agent team itself: how it decides what to cover, how it grounds ambiguous language, and how it keeps a suite green as the product changes. This is the area that most differentiates the product and where trust is won or lost. Grounded in the explore-agent page, memory page, and the how-Momentic-works agent-loop/auto-heal/step-cache material.

“Momentic is an AI testing platform that builds, runs, and maintains tests as your product changes.” momentic.ai

Mapped capabilities

4 capabilities

Explore agent from a diff
momentic ai explore diff [commitRange] and explore latest; mapping user-facing changes (including backend changes that surface to users) to journeys; editing an existing partially-covering test instead of adding a sibling; --dry-run discovery versus building; read-only with respect to application code.
Memory and disambiguation
Feeding past successful decisions to locator and assertion agents; per-org storage; 30-day inactivity expiry; writing memory on failed steps so real bugs fail consistently; per-test toggle and ai.useMemory default; skipped when caching is disabled.
Auto-heal and the step cache
The agent loop repairing steps as the UI drifts, and cached steps enabling fast reruns.
Bug discovery output
Potential product bugs surfaced during exploration with severity, ordered repro steps, and supporting evidence.

Illustrative example

A repo already contains checkout.test.yaml covering guest checkout through the payment confirmation screen. A PR adds a promo-code field to the checkout form. The user runs `momentic ai explore diff main..HEAD`. → The agent diffs the refs, identifies the promo-code field as a user-facing change, maps it to the checkout journey, detects that checkout.test.yaml already partially covers that journey, and proposes an edit to that existing test rather than authoring a new sibling test. It describes the observable behavior and coverage gap without proposing changes to the application's product code.

Execution environments and CLI

Where and how a test actually runs, and the setup path to get there — the surface a DevEx engineer owns. Grounded in the web quickstart, configuration docs, CLI reference, and the platform picker for web, iOS, and Android.

“Tests can be executed anywhere using the CLI : on your laptop, in a CI pipeline, on a cloud agent sandbox” momentic.ai

Mapped capabilities

4 capabilities

Onboarding and install paths
npx @momentic/wizard@latest, manual npm install --save-dev momentic plus momentic init, and the non-interactive coding-agent path; Node.js 22.12.0+/24.0.0+ prerequisite.
Auth and credential handling
Browser login writing ~/.momentic/auth.json, MOMENTIC_API_KEY precedence over that file, shell persistence, and CI secret configuration.
Local, CI, and cloud sandbox execution
Same tests run on a laptop, in a CI pipeline, or on a cloud agent sandbox; environment-dependent capabilities such as ffmpeg for local run-assertion frame sampling versus cloud-side evaluation.
Platform targets
Chromium-based web, iOS simulators and Android emulators without real devices, and React Native/Expo/Flutter projects tested via their native builds.

Assertions and application-under-test interactions

What a test can observe and manipulate in the app — the ceiling on what a team can actually verify. Grounded in the guides on authentication, page interactions, and visual testing, and the July 2026 run-assertions changelog entry.

Mapped capabilities

4 capabilities

Step assertions and run assertions
AI check and element check during a run versus plain-English run assertions evaluated after the run against sampled video frames; transient UI such as toasts, spinners, and pop-ups; any failed run assertion fails the run; video recording auto-enabled.
Authentication flows
Strategy choice across email OTP, SMS OTP, magic links, TOTP, cached sessions, and Vercel preview deployments.
Page interactions
File upload and download, dropdowns, and recording and validating network requests.
Visual testing
Visual diff snapshots stored in the repo alongside tests.

Illustrative example

A web test submits a form; the app shows a 'Saved successfully' toast for roughly 800ms. The test declares a run assertion: 'A success toast confirming the item was saved appeared at some point.' The test defines no step-level assertion for the toast. → Video recording is enabled automatically because the test declares a run assertion. The assertion is evaluated after the run completes against frames sampled from the run's video, not during step execution. If the toast never appeared, the run assertion fails and the overall run is marked failed even though every step passed.

Triage, gating, and flake control

What happens when a run goes red: classification, whether the pipeline blocks, and how a bad test is contained rather than left to erode trust in the suite. Grounded in the triage guide, quarantining in the dashboard description, and the editor/run-control changelog entries.

Mapped capabilities

4 capabilities

Failure classification during a run
Distinguishing product bugs from test defects while the run is in progress.
CI gating control
Deciding which failures block a merge and which are advisory.
Routing into AI maintenance
Sending classified failures back into agent-driven test repair.
Quarantine
Isolating known-bad or unstable tests via the cloud dashboard so they stop gating.

Results, analytics, and enterprise policy

The post-run and org-level surface: what evidence a run leaves behind, what a manager sees over time, and the controls a security reviewer requires. Grounded in the results and reporting docs, the dashboard description, and the published pricing tiers.

Mapped capabilities

4 capabilities

Run artifacts and reports
Runs, videos, traces, local reports, JSON export, JUnit output, and self-hosted run results.
Dashboard analytics and knowledge base
Run viewing, analytics over time, and an AI-maintained knowledge base supplying terminology and flows to the agents.
Plans, credits, and quotas
Free (2,000 credits/month, ~200 runs, 30-day retention), Pay-as-you-go ($125/month, 10,000 credits, no hard cap, 5 Android + 5 iOS devices, 5 SMS/OTP phone numbers), and Enterprise test-based custom pricing.
Identity, audit, and retention controls
SAML SSO, SCIM provisioning, audit log, unlimited mobile session length, and results-retention windows by tier.

Coverage is mapped from Momentic's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Momentic test?+

The coverage map above is generated from Momentic's public product surface: 6 scoring areas spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Momentic evals scored?+

Every eval set is graded the same way: pass/fail checks plus an LLM judge scoring 1–5 against each case's expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Momentic library include?+

The full Momentic library is built on request. The coverage map spans 6 areas and 24 capabilities; each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Momentic or my own agent?+

Request the library with your work email above. We'll build it out and set it up in a Corsac workspace, where you can run every test case against Momentic or your own agent with your own data.