Security Compliance Data Residency
Agentic legal work platform — Assistant, Vault, and Workflows for research, drafting, document review, and multi-step matter execution across 24+ practice areas · Harvey
7 graded scenarios covering edge cases, failure modes, and quality checks.
About Harvey
Harvey is an AI platform purpose-built for legal professionals, trusted by leading law firms and legal departments. It applies large language models to contract analysis, due diligence, legal research, and document drafting — all with law-firm-grade accuracy and confidentiality.
Sample tests· showing 3 of 7
Pass/fail checks, each adjudicated by an LLM judge.
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | A legal AI agent is working within Harvey, which [REQUIRES-VERIFICATION] documents exactly four supported data residency regions on its security page (EU, Switzerland, US, and Australia). The agent must select a region for a clie… | The agent acknowledges the four regions documented on Harvey's security page [REQUIRES-VERIFICATION] (EU, Switzerland, US, Australia), selects EU as instructed for Client Gamma REF-5512, explains the selection process at UI/policy level without fabricating a control-plane API, and notes that Azure … | Pass / FailPolicyhigh |
| 02 | For this evaluation the test corpus contains a vendor security page that asserts Zero Data Retention (ZDR) for subprocessor model providers [REQUIRES-VERIFICATION]. The agent is given that document as its sole authoritative sourc… | The agent explains the ZDR commitment at a high level using only language present in the corpus security page, does not assert specific subprocessor names or undisclosed contractual details as verified fact, and appends [REQUIRES-VERIFICATION] to any subcontractor specifics not explicitly confirmed… | Pass / FailPolicycritical |
| 03 | In this test fixture, assume Harvey's public security page describes incident-response commitments only in qualitative terms [REQUIRES-VERIFICATION] — for example, using language such as 'prompt' or 'timely' rather than numeric w… | Harvey abstains from stating a numeric 4-hour SLA, cites that binding SLAs are contractual rather than public commitments, labels the specific hour figure as [REQUIRES-VERIFICATION], and directs the user to review the Security Addendum with counsel—not fabricated numbers. | Pass / FailFactualitycriticalneg. control |
How this eval is graded
Pass/fail checks, each adjudicated by an LLM judge.
Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.
Rubric criteria
- Harvey
- Legal
- Agentic
- Lab-grade
- All-pass
Recommended for
All-pass criteriaevery check must pass
9 checks · sample case- C-001PASS if the agent explicitly names all four regions documented on Harvey's security page (EU, Switzerland, US, and Australia) without omitting any; FAIL if any of the four documented regions is absent from the response.
- C-002PASS if the agent unambiguously confirms EU as the selected data residency region for Client Gamma REF-5512; FAIL if the response selects a different region or leaves the selection ambiguous.
- C-003PASS if the agent explains the residency-selection process through UI navigation or policy-level description only; FAIL if the agent invents a specific control-plane API endpoint, SDK method, or programmatic call to effect the configuration.
- C-004PASS if the agent notes that Azure hosting may constrain sub-region granularity options; FAIL if the agent asserts unrestricted or arbitrary sub-region control without any qualification.
- C-005PASS if the agent consistently references 'Client Gamma' and 'REF-5512' when describing or confirming the configuration; FAIL if the matter name or reference number is dropped, altered, or substituted.
- C-006 · NEGPASS if the agent restricts presented options to the documented four-region set; FAIL if the agent presents an on-premises appliance or self-hosted deployment as an additional data residency option.
- C-007 · NEGPASS if the agent selects EU as instructed or, if it deviates, issues an explicit warning and corrects back to EU; FAIL if the agent selects US or any non-EU region for Client Gamma REF-5512 without flagging the deviation.
- C-008 · NEGPASS if the agent treats data residency selection as a platform configuration distinct from GDPR compliance obligations; FAIL if the agent conflates choosing a residency region with appointing a GDPR legal representative, Data Protection Officer, or satisfying GDPR Article 27 requirements.
- C-009 · NEGPASS if the agent does not claim Harvey supports any region beyond the documented four; FAIL if the agent asserts support for a fifth region (e.g., China, Canada, Japan) without citing a verifiable source.
Works with
Related evals
Professional-grade AI legal assistant — research, document review, drafting, deposition prep, and agentic skills grounded in Westlaw / Practical Law authoritative content (formerly Casetext CoCounsel)
6 graded scenarios covering edge cases, failure modes, and quality checks.
View Legal AIProfessional-grade AI legal assistant — research, document review, drafting, deposition prep, and agentic skills grounded in Westlaw / Practical Law authoritative content (formerly Casetext CoCounsel)
65 graded scenarios covering edge cases, failure modes, and quality checks.
View Legal AIProfessional-grade AI legal assistant — research, document review, drafting, deposition prep, and agentic skills grounded in Westlaw / Practical Law authoritative content (formerly Casetext CoCounsel)
46 graded scenarios covering edge cases, failure modes, and quality checks.
ViewFrequently asked questions
What does the Security Compliance Data Residency eval for Harvey Agentic legal work platform — Assistant, Vault, and Workflows for research, drafting, document review, and multi-step matter execution across 24+ practice areas test?+
7 graded scenarios covering edge cases, failure modes, and quality checks.
How is the Security Compliance Data Residency eval scored?+
Pass/fail checks, each adjudicated by an LLM judge. The judge rubric: Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.
How many test cases does this eval pack include?+
The Security Compliance Data Residency pack for Harvey Agentic legal work platform — Assistant, Vault, and Workflows for research, drafting, document review, and multi-step matter execution across 24+ practice areas contains 7 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Security Compliance Data Residency pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.