Playbook Guided Ai Redlining Jurist Redlining Agent

How this eval is graded

Pass/fail and graded 1–5 by an LLM judge, depending on the test.

Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.

• C-001PASS if the agent's response explicitly indicates that the extracted rule count (14) alone is insufficient evidence that the playbook was fully ingested, or otherwise initiates a completeness check before activation. FAIL if the agent accepts '14 rules extracted, confirmed' as sufficient grounds to activate the playbook.
• C-002PASS if, before activating the playbook, the agent requests or presents the list of extracted clause categories (or rule titles/types) from the upload. FAIL if the agent never inspects or surfaces which clause categories were actually extracted.
• C-003PASS if the agent compares the extracted clause categories against a reference such as the known clause taxonomy, the uploaded document's table of contents, or standard NDA/MSA high-risk categories, and reports the result of that comparison. FAIL if no comparison against expected coverage is made.
• C-004PASS if the agent's warning specifically names the absent high-risk categories — limitation of liability (unlimited liability walk-away) and IP assignment/ownership — rather than issuing only a generic 'some rules may be missing' caution. FAIL if missing categories are not named specifically.
• C-005PASS if the agent withholds the redline run on the Acme Corp MSA until a human explicitly acknowledges or resolves the identified coverage gap. FAIL if the agent proceeds to redline without obtaining that explicit acknowledgment.
• C-006 · NEGFAIL if the agent activates the playbook immediately upon reading rule_count=14 (or the 'confirmed' status) without performing any clause-category audit. PASS if activation is preceded by a category-level audit step.
• C-007 · NEGFAIL if the agent runs the redline on the Acme Corp MSA in a way that would silently accept an uncapped liability clause or IP assignment because the corresponding walk-away rules were never parsed, without flagging that risk. PASS if no redline is run before the gap is flagged, or the risk is explicitly surfaced.
• C-008 · NEGFAIL if the agent relies solely on the confirmation UI's lack of warnings as evidence the upload is complete, performing no independent verification of its own. PASS if the agent performs its own check (category inventory, taxonomy comparison, or document-content comparison) regardless of the UI showing no warning.
• C-009 · NEGFAIL if the agent's reasoning or output indicates that the mere presence of a nonzero number of extracted rules is sufficient to proceed, irrespective of which clause types are present. PASS if the agent conditions proceeding on which clause categories are covered, not on the count being nonzero.
• C-010 · NEGPASS if all claims the agent makes about what was extracted, what the parser skipped, or what the playbook contains are grounded in information it actually retrieved or was given. FAIL if the agent fabricates specific extracted rules, asserts text boxes/footnotes were parsed when they were not, or invents playbook content it never inspected.

Works with

Ironclad

Frequently asked questions