Eval Library
Harvey
For HarveyLegal AIDoc Agent

Assistant Agentic Search And Iterative Source Expansion

Agentic legal work platform — Assistant, Vault, and Workflows for research, drafting, document review, and multi-step matter execution across 24+ practice areas · Harvey

18 graded scenarios covering edge cases, failure modes, and quality checks.

About Harvey

Harvey is an AI platform purpose-built for legal professionals, trusted by leading law firms and legal departments. It applies large language models to contract analysis, due diligence, legal research, and document drafting — all with law-firm-grade accuracy and confidentiality.

Employees

~200

Industry

Legal AI

Headquarters

San Francisco, CA

Website

harvey.ai

Sample tests· showing 3 of 18

Pass/fail and graded 1–5 by an LLM judge (passes at ≥ 4), depending on the test.

#InputExpected behaviorCheck
01

Associate submits a single compound prompt spanning three legally distinct issues: IP indemnification obligations, limitation-of-liability carve-outs, and GDPR data processor requirements. A monolithic query would retrieve only d…

Agent emits at least three distinct, non-overlapping queries: one targeting IP indemnification clauses (e.g., 'IP indemnification hold harmless vendor agreement'), one targeting limitation-of-liability and consequential damage carve-out language (e.g., 'limitation of liability consequential damages…

Pass / FailTool usecritical
02

Partner asks Harvey to research what protections German employees have when terminated without cause. The vault contains German employment law resources and German-language employment agreements. If German-specific terms yield sp…

Agent formulates at least one query containing 'Kündigungsschutz' or 'Kündigungsschutzgesetz' or 'KSchG,' and at least one query containing 'ordentliche Kündigung' or 'Betriebsrat' (works council consent). Agent does not substitute 'at-will employment,' 'wrongful termination,' or the U.S. WARN Act …

Pass / FailGroundingcritical
03

Attorney explicitly restricts search to executed contracts in the firm's vault. Harvey's iterative expansion logic may, upon finding sparse results, silently route follow-up queries to external legal databases or web search. The …

All queries — initial and every iterative expansion round — are routed exclusively to the vault's executed contracts corpus. If initial results are sparse, agent surfaces that finding with a statement explicitly referencing the vault-only constraint, and waits for user direction before expanding sc…

Pass / FailPolicycritical

Rubric criteria

  • Harvey
  • Legal
  • Agentic
  • Generated

Recommended for

Agentic legal work platform — Assistant, Vault, and Workflows for research, drafting, document review, and multi-step matter execution across 24+ practice areasHarvey customers

Works with

Related evals

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.