Eval directory
Evals for Harvey
10 evaluation packs covering adversarial robustness, safety gates, workflow quality, and operator-level checks for Harvey AI products.
About Harvey
Harvey is an AI platform purpose-built for legal professionals, trusted by leading law firms and legal departments. It applies large language models to contract analysis, due diligence, legal research, and document drafting — all with law-firm-grade accuracy and confidentiality.
Available eval packs for Harvey
10 packs ready to run.
Assistant Agentic Search And Iterative Source Expansion
18 graded scenarios covering edge cases, failure modes, and quality checks.
Assistant Citation Grounded Q A
Answer Relevance7 graded scenarios covering edge cases, failure modes, and quality checks.
Assistant Conversational Q A And Prompt Entry
78 graded scenarios covering edge cases, failure modes, and quality checks.
Assistant Deep Analysis And Long Form Memo Generation
54 graded scenarios covering edge cases, failure modes, and quality checks.
Confidentiality Privilege Tenant Isolation
PII Leakage5 graded scenarios covering edge cases, failure modes, and quality checks.
Contract Review Negotiation Intelligence
10 graded scenarios covering edge cases, failure modes, and quality checks.
Legal Drafting Safety Anti Fabrication
Hallucination10 graded scenarios covering edge cases, failure modes, and quality checks.
Security Compliance Data Residency
7 graded scenarios covering edge cases, failure modes, and quality checks.
Vault Review Tables Matter Retrieval
Answer Relevance9 graded scenarios covering edge cases, failure modes, and quality checks.
Workflow Agents Governance
6 graded scenarios covering edge cases, failure modes, and quality checks.
Why eval Harvey AI
Harvey's AI features ship behind brand promises about accuracy, safety, and reliability. Buyers and integrators need to know those promises hold up under adversarial prompts, edge-case workflows, and the long tail of real customer inputs — not just the demo path.
The Corsac eval library for Harvey measures four dimensions teams care about most when deploying legal ai agents:
- Adversarial robustness — does the agent resist prompt injection, jailbreaks, and social-engineering attempts?
- Workflow quality— does it complete the task buyers were shown in the demo, on inputs that don't look like the demo?
- Safety gates — does it escalate or refuse when it should, and only then?
- Operator quality — does it preserve analyst trust by surfacing the right context at the right time?
Every eval pack above is hand-authored against Harvey's public product surface and runnable in Corsac with your own data.