Vault Review Tables Matter Retrieval

How this eval is graded

Pass/fail checks, each adjudicated by an LLM judge.

Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.

• C-001PASS if the output contains exactly three distinct labeled columns — change-of-control definition, consent requirement, and termination notice period — as separate fields in a structured table; FAIL if any of the three columns is absent, merged with another, or replaced by a prose narrative.
• C-002PASS if each of the 48 acquisition agreements in Matter REF-2041 is represented by its own dedicated row in the table; FAIL if any two or more agreements are collapsed into a single row or if the row count is fewer than the number of processed documents.
• C-003PASS if every populated cell contains a verifiable citation (document identifier and section or page reference) for the extracted value; FAIL if any populated cell presents a value without an accompanying source citation.
• C-004PASS if every cell where the relevant clause is not present in the source document is explicitly marked 'N/A' or an equivalent unambiguous absence marker; FAIL if any such cell is left blank or silently omitted.
• C-005PASS if agreements whose change-of-control definitions materially differ from one another are individually recorded with a flag or annotation noting the conflict; FAIL if differing definitions are silently synthesized into a single harmonized value without disclosure.
• C-006 · NEGPASS if the change-of-control column contains only text sourced from each agreement's change-of-control definition provision; FAIL if termination clause language or any other clause's text appears in the change-of-control column for any row.
• C-007 · NEGPASS if every agreement that requires board consent has that requirement recorded in its consent-requirement cell; FAIL if board consent is omitted or downgraded to a generic consent label in any row without an explanatory flag.
• C-008 · NEGPASS if any row whose source document could not be fully parsed due to OCR or extraction failure carries an explicit error indicator (e.g., '[EXTRACTION ERROR]') in affected cells; FAIL if such rows are silently dropped from the table or left blank without an error flag.
• C-009 · NEGPASS if all cell values are directly traceable to extracted text in the source document; FAIL if any cell contains a value that was not present in the source document and was instead generated or inferred without disclosure.
• C-010 · NEGPASS if the output is a structured tabular representation with rows and columns; FAIL if a blended narrative or executive summary replaces or substitutes for the structured table.

Works with

Harvey

Frequently asked questions