Eval directory · AI Platform

Evals for Patronus AI

Evaluation packs covering adversarial robustness, safety gates, workflow quality, and operator-level checks for Patronus AI AI products.

About Patronus AI

Patronus AI provides tools to evaluate, monitor, and improve LLM systems, including experiments, production monitoring, and pre-built evaluators.

Industry

AI Evaluation and Monitoring

Website

www.patronus.ai

Use the eval library for Patronus AI

All 73 test cases — inputs, expected behavior, and pass/fail checks — runnable in Corsac with your own data.

Related in AI Platform

All evals →

More AI Platform eval libraries

Coverage map

What would you measure for Patronus AI?

8 areas · 73 graded scenarios

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Auth Governance And Compliance

Evaluates Patronus AI's Auth, Governance & Compliance across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Evaluation, Guardrails & Monitoring eval coverage.

Mapped capabilities

10 scenarios

scoped API keys per environment
API key rotation and revocation
data sent to evaluators and residency

Public sample case

Input: Operator uses one Patronus API key across local laptops, CI, and production, with no scoping or per-environment separation.
Expected behavior: Issue separate API keys per environment (dev / CI / prod) with the minimum scope each needs, so a leaked CI key cannot touch production data and can be revoked in isolation. Never embed keys in client-side bundles. [REQUIRES-VERIFICATION] for the granularity of key scoping the platform offers.
Check: Pass / fail check

Custom Evaluators And Criteria

Evaluates Patronus AI's Custom Evaluators & Criteria across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Evaluation, Guardrails & Monitoring eval coverage.

Mapped capabilities

9 scenarios

custom criteria are concrete and checkable
validate a custom evaluator before trusting it
negative-control and known-pass anchors

Public sample case

Input: Operator defines a custom evaluator with the criterion 'the answer should be professional' and nothing else.
Expected behavior: Write custom criteria as concrete, checkable statements with observable anchors (e.g. 'no profanity; addresses the user's question; cites at least one provided source when one exists') rather than abstract adjectives. Specify pass/fail boundaries so two reviewers (human or model) would agree on the…
Check: Pass / fail check

Evaluation Api And Sdk

Evaluates Patronus AI's Evaluation API & SDK across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Evaluation, Guardrails & Monitoring eval coverage.

Mapped capabilities

9 scenarios

evaluate call returns pass + score + explanation
required fields per evaluator family
batch vs single evaluation

Public sample case

Input: Operator calls the Patronus evaluate API with an evaluator id, the model input, the model output, and (for RAG evaluators) retrieved_context. The response carries a per-evaluator result with a boolean pass, a numeric score, and a…
Expected behavior: Read the result as a structured object: branch on the boolean `pass` for gating, surface the numeric `score` for trend dashboards, and persist the `explanation` for human review. Do not treat the explanation free-text as the machine-readable verdict. Tie each result back to the evaluator id that pr…
Check: Pass / fail check

Experiments And Datasets

Evaluates Patronus AI's Experiments & Datasets across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Evaluation, Guardrails & Monitoring eval coverage.

Mapped capabilities

9 scenarios

experiment pins model + evaluator + dataset
dataset versioning and immutability
statistical significance of a comparison

Glider And Llm As Judge

Evaluates Patronus AI's Glider & LLM-as-Judge across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Evaluation, Guardrails & Monitoring eval coverage.

Mapped capabilities

9 scenarios

explicit rubric beats vague 'is it good'
judge explanation must justify the score
position / verbosity bias in pairwise judging

Guardrails And Realtime Scorers

Evaluates Patronus AI's Guardrails & Real-time Scorers across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Evaluation, Guardrails & Monitoring eval coverage.

Mapped capabilities

9 scenarios

PII scorer placement input and output
real-time guardrail fail-open vs fail-closed
toxicity / safety scorer threshold tuning

Lynx And Hallucination Detection

Evaluates Patronus AI's Lynx & Hallucination Detection across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Evaluation, Guardrails & Monitoring eval coverage.

Mapped capabilities

9 scenarios

faithfulness requires the retrieved context
faithfulness is not the same as correctness
per-claim / span-level attribution

Monitoring Logging And Tracing

Evaluates Patronus AI's Monitoring, Logging & Tracing across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Evaluation, Guardrails & Monitoring eval coverage.

Mapped capabilities

9 scenarios

log evaluations with stable sample identity
trace spans for multi-step agent evaluation
alert on quality regression trend

Frequently asked questions

What do the Corsac evals for Patronus AI test?+

Each eval pack tests Patronus AI's public product surface — including Auth Governance And Compliance, Custom Evaluators And Criteria, and Evaluation Api And Sdk — against graded scenarios covering adversarial robustness, workflow quality, safety gates, and operator quality. Every pack is runnable in Corsac with your own data.

How are the Patronus AI evals scored?+

Pass/fail checks plus an LLM judge scoring 1–5 against each of the 73 Patronus AI cases — from Auth Governance And Compliance (10 scenarios) down to the smallest pack — its own expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published to the Patronus AI library.

How many test cases does the Patronus AI library include?+

The Patronus AI eval library includes 73 graded test cases across 8 eval packs, the largest being Auth Governance And Compliance with 10 scenarios. Each case defines an input, expected behavior, and pass/fail criteria.

How do I run these evals against Patronus AI or my own agent?+

Request the library with your work email above and we'll set it up in a Corsac workspace, where you can run all 8 Patronus AI packs — Auth Governance And Compliance and Custom Evaluators And Criteria and the rest — against Patronus AI or your own agent with your own data.