Eval directory

Evals for Cleric

Mapped eval coverage for Cleric — adversarial robustness, safety gates, workflow quality, and operator-level checks across its public product surface.

Use the eval library for Cleric

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Cleric?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Change Verification

Following a pull request from open to merged to deployed, recording the expected outcome, and checking production behavior against real traffic over a verification window.

“Cleric tracks each PR from open to deployed, then checks that production behaves as expected.” cleric.ai

Mapped capabilities

4 capabilities

PR-to-production tracking
Associating an open PR with its merge, its deployed revision, and the point at which the new code is serving traffic.
Expected-outcome recording
Capturing what the change is supposed to do at PR time so later production behavior can be judged against it.
Regression detection against real traffic
Deciding pass/regression from live production signals rather than pre-merge test results.
Verification window and re-check cadence
Running checks repeatedly across a multi-day window and reporting progress and next-run timing.

Illustrative example

A PR titled "checkout-api: Tighten timeout handling" is opened at 09:12, merged at 09:41, and reaches production at 09:44 with a new revision serving traffic. Ask for the current verification status of this change at day 3 of the window. → The response reports verification as still in progress against real production traffic rather than complete, states that checks are currently passing with no regression detected, and indicates that another check is scheduled. It does not treat reaching production or a passing first check as final sign-off.

Investigation and Root Cause

Opening an investigation from a production signal, testing candidate causes against production systems, and returning an actionable finding with a proposed fix.

“Cleric groups alerts that share a root cause into a single investigation.” cleric.ai

Mapped capabilities

4 capabilities

Multi-trigger intake
Starting the same investigation flow from a change regression, alert, support ticket, engineer request, scheduled check, customer report, or manual review.
Alert grouping by shared cause
Collapsing multiple alerts that share a root cause into a single investigation rather than one per alert.
Hypothesis testing against production
Evaluating candidate causes using live production systems and reporting which were ruled in or out.
Fix proposal and handoff
Producing a proposed fix and an investigation record an engineer can act on or sign off.

Illustrative example

23 separate alerts fire within a few minutes across services downstream of payment-api, all consistent with a p99 latency regression in payment-api. Ask how many investigations should be open and what each covers. → The response opens a single investigation scoped to the payment-api p99 latency regression, states the number of alerts grouped under it, and identifies the affected downstream services as symptoms of that one investigation rather than as separate issues.

Integrations and Signal Surface

Connecting to the observability, paging, code, and chat systems that supply signals and carry Cleric's output back to the team.

Mapped capabilities

4 capabilities

Observability and paging intake
Ingesting signals from Datadog and PagerDuty as investigation triggers.
Source control integration
Reading GitHub PR and merge state to drive change verification.
Chat-surface delivery and commands
Surfacing investigations in Slack and exposing Slash Commands to engineers.
Custom Agents and Scheduled Monitors
Team-defined agents and recurring checks that open investigations on a schedule.

Operational Memory and Learning

Persisting engineering judgment across investigations so outcomes from one incident inform the next, backed by a knowledge graph and verification engine.

Mapped capabilities

4 capabilities

Knowledge graph of the production system
Retaining structure and relationships discovered during investigations for reuse.
Learning from investigation outcomes
Incorporating whether a diagnosis and fix actually held after the fact.
Fix verification against production
Distinguishing a correct diagnosis from a problem that is actually fixed, using production as ground truth.
Trace quality and regression evaluation
Ranking investigation traces by pairwise comparison to catch quality regressions over time.

Commercial Model and Credits

The per-resolved-Issue credit plan: a monthly credit pool spent on Issues and Custom Work, with a locked rate and a pilot before signing.

“Up to 100 Issues per month, or any mix of Issues and Custom Work.” cleric.ai

Mapped capabilities

4 capabilities

Credit accounting for Issues and Custom Work
Debiting 10 credits per Issue and 20 per Custom Work item against a monthly pool at $2 per credit.
Plan limits and mix
Team plan at $2,000/month and 1,000 credits supporting up to 100 Issues per month, or any mix.
Rate stability over time
Holding the per-Issue price flat regardless of how long the work takes or what models are used.
Pilot and onboarding path
A 14-day pilot on the customer's own production inside their existing on-call rotation, set up in under a day.

Access Posture and Trust Boundaries

How the agent reaches production and private systems and what it is permitted to do there, as stated on the public surface.

“Read-only by default · SOC 2 Type II” cleric.ai

Mapped capabilities

4 capabilities

Read-only default
Operating read-only by default when touching production systems.
Private network access
Reaching private resources inside a customer VPC the way a human engineer would.
Authentication and tenancy controls
SSO via Google Workspace on Team; OIDC and SAML on Enterprise.
Auditability and compliance artifacts
Audit logs, SOC 2 Type II, and a signed DPA.

Coverage is mapped from Cleric's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Cleric test?+

The coverage map above is generated from Cleric's public product surface: 6 scoring areas spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Cleric evals scored?+

Every eval set is graded the same way: pass/fail checks plus an LLM judge scoring 1–5 against each case's expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Cleric library include?+

The full Cleric library is built on request. The coverage map spans 6 areas and 24 capabilities; each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Cleric or my own agent?+

Request the library with your work email above. We'll build it out and set it up in a Corsac workspace, where you can run every test case against Cleric or your own agent with your own data.