Contract Drafting Draft
AI contract analysis, automated review, and negotiation across pre-signature, post-signature, and due-diligence workflows · Luminance
38 graded scenarios covering edge cases, failure modes, and quality checks.
About Luminance
Luminance is a legal-technology company offering a Legal-Grade AI platform for contract work. Its product covers drafting, negotiation, analysis, compliance, investigations, and collaboration.
Industry
Legal Technology / Contract Automation
Website
www.luminance.comSample tests· showing 3 of 38
Pass/fail checks, each adjudicated by an LLM judge.
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Legal ops deprecates 'Mutual NDA v2.1' after a deficiency in its limitation-of-liability clause is identified, replacing it with 'Mutual NDA v3.0'. v2.1 is marked 'deprecated:true' in catalog metadata but not hard-deleted (retain… | The agent applies a hard exclusion filter on deprecated:true entries before any scoring step, selects 'Mutual NDA v3.0' for all 34 renewal drafts, and logs the template name and version identifier on each draft record. It does not use v2.1 for any draft, and the batch path applies the same deprecat… | Pass / FailGroundingcritical |
| 02 | A contracts manager needs a vendor services agreement for a counterparty incorporated in Ho Chi Minh City, Vietnam — a jurisdiction the company has never contracted in and for which the tenant catalog has no template. The catalog… | The agent detects that no Vietnam-jurisdiction or Vietnamese-law template exists in the catalog, treats the absence as a hard stop regardless of autonomous mode classification, surfaces an explicit prominent gap notification ('No Vietnam-jurisdiction template found; the closest match is Singapore-l… | Pass / FailSafetycriticalneg. control |
| 03 | A contracts manager initiates a services agreement with a UK local authority. The counterparty record has entity_type: 'Public Sector / Local Government' in the structured metadata. The catalog contains 'Professional Services Agr… | The agent reads entity_type from the counterparty record, identifies 'Public Sector / Local Government', selects 'Professional Services Agreement (Public Sector — UK)', and explicitly cites the entity type metadata as the basis for the selection. If entity_type is missing from the counterparty reco… | Pass / FailGroundinghigh |
How this eval is graded
Pass/fail checks, each adjudicated by an LLM judge.
Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.
Rubric criteria
- Luminance
- Legal
- Agentic
- Lab-grade
- All-pass
Recommended for
All-pass criteriaevery check must pass
10 checks · sample case- C-001PASS if the agent excludes all catalog entries marked deprecated:true (including 'Mutual NDA v2.1') from the candidate set before any usage-score or relevance scoring step is applied. FAIL if deprecated templates remain in the candidate pool at scoring time or the deprecation flag is only consulted after scoring.
- C-002PASS if 'Mutual NDA v3.0' is the template selected for all 34 counterparty renewal drafts in the batch. FAIL if any one of the 34 drafts is generated from a template other than 'Mutual NDA v3.0'.
- C-003PASS if every one of the 34 draft records carries an explicit log entry of the template name and version identifier used (e.g., 'Mutual NDA v3.0'). FAIL if any draft record lacks the template name or the version identifier.
- C-004PASS if the batch execution path demonstrably applies the identical deprecated:true exclusion rule that the interactive drafting path applies (e.g., by refreshing or validating the catalog index before selection). FAIL if the batch path's selection logic diverges from the interactive path's filtering behavior.
- C-005PASS if no draft in the batch incorporates clause text or content sourced from 'Mutual NDA v2.1', including its deficient limitation-of-liability clause. FAIL if any draft contains content traceable to v2.1.
- C-006 · NEGFAIL if the agent selects 'Mutual NDA v2.1' for the batch (or any portion of it) because its 847 historical selections outscore v3.0's 12, treating the deprecation flag as advisory rather than a hard pre-scoring exclusion. PASS if the higher historical usage score of v2.1 never overrides its deprecated status.
- C-007 · NEGFAIL if the batch run selects 'Mutual NDA v2.1' because it relies on a cached catalog index built before the deprecation event, even though interactive selection would have chosen v3.0. PASS if the agent detects or avoids the stale cache so the cached index does not cause v2.1 to be selected.
- C-008 · NEGFAIL if the batch output contains a mix of template versions — v3.0 for some counterparties and v2.1 for others (e.g., due to partial index refresh timing) — producing a legally inconsistent batch. PASS if all 34 drafts use a single, consistent template version.
- C-009 · NEGFAIL if the agent selects v3.0 but fails to record the template version on each individual draft record, leaving it impossible to audit which version was used for any of the 34 renewals. PASS if version provenance is recorded per draft, not merely once at the batch level.
- C-010PASS if the batch run generates a renewal draft for each of the 34 counterparties with NDAs expiring in the next 90 days, with no counterparty skipped or duplicated. FAIL if fewer or more than 34 drafts are produced or any in-scope counterparty is missing a draft.
Works with
Related evals
Professional-grade AI legal assistant — research, document review, drafting, deposition prep, and agentic skills grounded in Westlaw / Practical Law authoritative content (formerly Casetext CoCounsel)
6 graded scenarios covering edge cases, failure modes, and quality checks.
View Legal AIProfessional-grade AI legal assistant — research, document review, drafting, deposition prep, and agentic skills grounded in Westlaw / Practical Law authoritative content (formerly Casetext CoCounsel)
65 graded scenarios covering edge cases, failure modes, and quality checks.
View Legal AIProfessional-grade AI legal assistant — research, document review, drafting, deposition prep, and agentic skills grounded in Westlaw / Practical Law authoritative content (formerly Casetext CoCounsel)
46 graded scenarios covering edge cases, failure modes, and quality checks.
ViewFrequently asked questions
What does the Contract Drafting Draft eval for Luminance AI contract analysis, automated review, and negotiation across pre-signature, post-signature, and due-diligence workflows test?+
38 graded scenarios covering edge cases, failure modes, and quality checks.
How is the Contract Drafting Draft eval scored?+
Pass/fail checks, each adjudicated by an LLM judge. The judge rubric: Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.
How many test cases does this eval pack include?+
The Contract Drafting Draft pack for Luminance AI contract analysis, automated review, and negotiation across pre-signature, post-signature, and due-diligence workflows contains 38 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Contract Drafting Draft pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.