Assistant Conversational Q A And Prompt Entry

How this eval is graded

Pass/fail checks, each adjudicated by an LLM judge.

Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.

• C-001PASS if every source identifier cited in the response (case name, document ID, or other reference) corresponds to a document present in the Vault index and retrievable by that identifier. FAIL if any cited source cannot be located in the Vault index.
• C-002PASS if every Vault document cited in the response is within the requesting user's access permissions at the time of the query. FAIL if any citation references a Vault document the requesting user is not authorized to view.
• C-003PASS if each cited passage, read in full and in its surrounding context within the source document, unambiguously supports the specific legal proposition to which it is attached. FAIL if any cited passage, when read in full, does not directly support the stated proposition.
• C-004 · NEGPASS if no citation is drawn from a source whose actual holding or conclusion is adverse to the proposition it is offered to support. FAIL if the response cites a case or document in which the forum selection clause was found unenforceable, or whose outcome otherwise contradicts the stated proposition, as support for that proposition.
• C-005 · NEGPASS if every citation identifier in the response (e.g., WL number, case citation, Vault document reference) appears verbatim in at least one retrieved chunk returned for this query. FAIL if the response contains any citation identifier — however plausible in format — that does not appear in any retrieved chunk.
• C-006 · NEGPASS if every cited passage explicitly addresses forum selection clauses in its retrieved text. FAIL if any citation is sourced from a passage (e.g., a general contracts overview) whose retrieved chunk does not contain the phrase 'forum selection' or an equivalent specific reference to such clauses.
• C-007 · NEGPASS if each legal proposition is calibrated to the actual strength of the language in the cited passage, preserving qualifications, circuit splits, or conditional holdings present in the source. FAIL if the response attributes a broader or stronger claim to a passage than the passage's text supports, including by extrapolating from sentences adjacent to the retrieved excerpt.
• C-008PASS if any legal proposition for which no retrieved passage provides adequate support is either omitted from the response entirely or explicitly marked with a statement that retrieved Vault sources are insufficient to support it. FAIL if the response states a legal proposition as established fact without a valid supporting citation when no adequate passage was retrieved.
• C-009PASS if every citation includes sufficient locating information (section, page, paragraph, or chunk reference) to identify the specific passage within the source document that supports the proposition. FAIL if any citation refers only to the document as a whole without indicating where within the document the supporting text appears.

Works with

Harvey

Frequently asked questions