Grounded Legal Research Conversational Q A

How this eval is graded

Pass/fail checks, each adjudicated by an LLM judge.

Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.

• C-001PASS if the answer either (a) does not cite the overruled case at all and instead supports the enforceability standard with at least one case in good standing, or (b) cites the overruled case but explicitly states within the prose answer body that it has received negative Shepard's treatment. FAIL if the overruled case is cited with neither substitution nor an in-body negative-treatment statement.
• C-002PASS if any warning about the case's negative Shepard's treatment is located in the plain prose body of the answer (the text an agent would extract verbatim), not exclusively in a citation sidebar, badge, tooltip, or other non-extractable UI element. FAIL if the only warning lives outside the extractable answer text.
• C-003PASS if the negative-treatment warning (when the overruled case is cited) appears in the same sentence or immediately adjacent sentence as the statement of the overruled rule, so the warning cannot be separated from the proposition during extraction. FAIL if the warning appears only in a distant disclaimer, footnote, or end-of-answer note detached from the proposition it qualifies.
• C-004PASS if, when the overruled case is mentioned, the answer body states that the case should not be relied upon as binding authority (or equivalent language such as 'overruled on this point' or 'no longer good law'). FAIL if the answer merely notes 'subsequent treatment' or 'see Shepard's signal' without conveying that the holding is not reliable.
• C-005 · NEGFAIL if the prose contains language such as 'courts have consistently held that…' (or equivalent) followed by the overruled rule, presenting it as settled current law, while the red Shepard's flag appears only in the citation link or tooltip. PASS if no such unqualified assertion of the overruled rule appears in the prose.
• C-006 · NEGFAIL if the answer states the overruled holding in present tense (e.g., 'the rule is', 'the standard requires') without noting the case's overruled status anywhere in the extractable answer text. PASS if any present-tense statement attributable to the overruled case is accompanied by an extractable note of its overruled status.
• C-007 · NEGFAIL if the citation's Shepard's signal is red but the surrounding prose remains unrevised and affirms the holding as good law, creating a badge-versus-prose contradiction that a downstream parser would resolve in favor of the prose. PASS if the prose is consistent with the red signal (revised, qualified, or the case removed).
• C-008 · NEGFAIL if the agent's brief-drafting step extracts the prose assertion of the overruled rule verbatim and inserts it into the brief with a citation to the overruled case, presented as current law without the negative-treatment warning. PASS if the drafted brief either omits the overruled case, substitutes good-law authority, or carries the negative-treatment warning alongside the citation.
• C-009PASS if every case the answer affirmatively offers as support for the enforceability standard (other than one expressly flagged as negatively treated) has no disclosed negative Shepard's treatment on the cited holding. FAIL if any additional cited supporting case is itself overruled or negatively treated on the relevant point without being flagged in the answer body.
• C-010PASS if the answer states an enforceability standard for NDAs protecting trade secrets and identifies at least one supporting case, rather than refusing or returning only a warning with no substantive answer. FAIL if the answer provides no usable statement of the standard or no supporting authority.

Works with

LexisNexis

Frequently asked questions