Retrieval Augmented Generation Pipeline Five Stage Prompt Checking

How this eval is graded

Pass/fail and graded 1–5 by an LLM judge, depending on the test.

Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.

• C-001PASS if the returned chunk contains a distinct label marking the enacted statutory text segment (e.g., '[Enacted statutory text — 42 U.S.C.A. § 1983]'). FAIL if the enacted statutory text appears without an explicit content-type label identifying it as enacted text.
• C-002PASS if every West editorial annotation (keynote, headnote, or editorial commentary) in the returned chunk carries a label identifying it as editorial material that is not statutory text (e.g., '[West editorial annotation — not statutory text]'). FAIL if any editorial annotation appears without such a label.
• C-003PASS if the statutory language the agent extracts for the brief is sourced exclusively from the segment labeled as enacted statutory text. FAIL if the extraction includes any content originating outside the labeled enacted-text segment.
• C-004PASS if the agent's response includes an explicit note that the U.S.C.A. is an annotated code containing West editorial material that is not part of the enacted statute. FAIL if no such disclosure appears anywhere in the response.
• C-005 · NEGPASS if no West keynote (e.g., a summary of a court holding) is presented or quoted in the agent's output as the text of 42 U.S.C.A. § 1983. FAIL if any keynote or case-holding summary is placed inside or adjacent to the enacted-text segment without a label, or is quoted in the brief-ready extraction as statutory language.
• C-006 · NEGPASS if the returned chunk preserves machine-readable content-type labels distinguishing enacted text from editorial material. FAIL if all content-type labels have been stripped such that enacted text and editorial annotations cannot be distinguished by label.
• C-007 · NEGPASS if the annotations returned are West U.S.C.A. annotations, or, if U.S.C.S. (LexisNexis) annotations are returned instead, the substitution is explicitly disclosed to the user. FAIL if U.S.C.S. annotations are returned in place of the requested U.S.C.A. annotations with no disclosure of the swap.
• C-008 · NEGPASS if the enacted statutory text and any keynotes in the returned chunk are separated by an explicit delimiter or labeled boundary. FAIL if enacted text and keynotes are concatenated as one undifferentiated block with no delimiter between them.
• C-009 · NEGPASS if the agent selects content for extraction based on its content-type labeling rather than its position in the chunk. FAIL if the agent uses the first substantive paragraph of the chunk as the statutory text regardless of whether that paragraph is labeled as enacted text or editorial material.
• C-010PASS if the retrieved chunk and the agent's extraction correspond to 42 U.S.C.A. § 1983 as requested. FAIL if the agent retrieves or extracts text attributed to a different statute or section.

Works with

LexisNexis

Frequently asked questions