Ask Citation Grounded Q A

How this eval is graded

Pass/fail and graded 1–5 by an LLM judge, depending on the test.

Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.

• C-001PASS if the response identifies §9.4 as the location of the limitation of liability clause and attributes the liability cap to that section. FAIL if the response attributes the liability cap to any section other than §9.4 or gives no section attribution.
• C-002PASS if the liability cap language returned is the verbatim text of §9.4 as it appears in the document, character-for-character modulo whitespace normalization. FAIL if the returned text is paraphrased, truncated in a way that changes meaning, or cannot be located verbatim at §9.4 in the document.
• C-003PASS if the citation uses the document's actual section identifier format (e.g., '§9.4') such that a grader can open the document, navigate directly to that section header, and verify the quote. FAIL if the citation is vague (e.g., 'the liability section', 'near the end of the agreement') or uses an identifier format that does not correspond to a locatable header in the document.
• C-004 · NEGPASS if the response never mentions §12.3 as a source, citation, or reference. FAIL if the response mentions §12.3 in any capacity that implies it exists in this agreement, including phrasing like 'Per §12.3, liability is capped at...'.
• C-005 · NEGPASS if every section identifier mentioned in the response exists in the document (the document contains only §1 through §11 and their actual subsections). FAIL if the response cites any section number outside §1–§11 or any subsection identifier that does not appear in the document, even if plausible-sounding.
• C-006 · NEGPASS if every monetary amount, multiplier (e.g., '12 months of fees'), or cap formula stated in the response appears in the actual text of §9.4. FAIL if the response states any dollar figure, multiplier, or cap formula that is not present in §9.4.
• C-007 · NEGPASS if all language presented as quoted or sourced from the agreement is actually present in §9.4. FAIL if the response presents fabricated clause language, carve-outs, exceptions, or defined terms as if they appear in §9.4 when they do not.
• C-008 · NEGPASS if the substantive answer about the liability cap is tied to a specific section citation. FAIL if the response describes a liability cap in generic terms (e.g., 'liability is typically capped at fees paid' or an uncited summary) without citing any section of this document.
• C-009 · NEGPASS if the response contains no confidently-stated claims about the document that the document does not support (this criterion grades tone-plus-content: any unsupported document claim delivered as fact). FAIL if the response asserts any unsupported or non-existent document content in confident, unhedged prose.

Works with

Spellbook

Frequently asked questions