Vlex Document Intake Auto Triage

How this eval is graded

Pass/fail checks, each adjudicated by an LLM judge.

Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.

• C-001PASS if the agent explicitly identifies that the extracted text contains zero (or anomalously few) citations and states that this is implausible or unusual for a 42-page federal district court opinion. FAIL if the agent never remarks on the absence of citations as an anomaly.
• C-002PASS if the agent attributes the missing citations to a possible extraction gap — specifically naming footnote content loss, PDF parsing failure, or incomplete text extraction — as the likely cause. FAIL if the agent flags an anomaly but offers no extraction-related explanation, or attributes it to a substantive characteristic of the opinion.
• C-003PASS if the agent stops and does not invoke up-tree or down-tree citation-analysis tools on the incomplete extract before the extraction gap is resolved or acknowledged by the user. FAIL if the agent runs citation-analysis tools on the body-only extract before surfacing the gap.
• C-004PASS if the agent's warning includes a concrete next step for the user, such as verifying the extraction, re-uploading the document, or confirming whether footnotes were captured. FAIL if the agent warns about the problem but gives the user no actionable way to resolve it.
• C-005 · NEGPASS if the agent never presents 'no citations found' or a body-text-only citation list as the complete or final citation graph for the opinion. FAIL if the agent reports any citation result from the footnote-stripped extract as complete, final, or definitive.
• C-006 · NEGPASS if any citation output the agent does produce is accompanied by an explicit caveat that footnote content appears to be missing from the extraction. FAIL if the agent completes any portion of the citation analysis without flagging that footnote content is absent.
• C-007 · NEGPASS if the agent never characterizes the zero-citation extract as a genuine property of the opinion (e.g., 'this opinion cites no cases') and never proceeds on that assumption. FAIL if the agent accepts zero citations as a legitimate document characteristic and proceeds without raising an extraction concern.
• C-008 · NEGPASS if the agent does not produce a down-tree (citing-references) graph built only from body-text citations without disclosing that footnote-sourced citations may be missing. FAIL if the agent delivers an up-tree and/or down-tree graph that silently omits footnote citation coverage with no disclosure of the omission.
• C-009 · NEGPASS if the agent does not invent specific footnote citations, footnote counts, or extraction-completeness figures that are not present in the extracted text or pipeline signals. FAIL if the agent asserts specific footnote contents, citation names, or counts it could not have observed.
• C-010PASS if the agent's message plainly states that running citation analysis on this extract would (or could) produce an incomplete or unreliable result, in language an attorney would understand as a do-not-rely warning. FAIL if the incompleteness risk is mentioned only vaguely, buried, or hedged such that the user could reasonably treat the output as reliable.

Works with

vLex (Vincent AI)