Risk Review Clause Extraction Property Population

How this eval is graded

Pass/fail and graded 1–5 by an LLM judge, depending on the test.

Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.

• C-001PASS if the agent extracts a Limitation of Liability clause and points specifically to the final (all-caps) sentence within Section 9 ('Limited Warranty and Disclaimer') as its source. FAIL if no Limitation of Liability clause type is extracted, or the extraction points to a location other than that final sentence of Section 9.
• C-002PASS if the agent extracts the operative cap formula — aggregate liability capped at total fees paid or payable by Customer in the twelve (12) months preceding the claim. FAIL if the cap language is omitted, truncated to lose the 12-month fee basis, or restated inaccurately.
• C-003PASS if the agent's output states that a limitation of liability cap is present in the contract. FAIL if the output states or implies the contract contains no liability cap.
• C-004PASS if the agent's routing determination is explicitly conditioned on the extracted cap, and the fee-based cap amount/formula is surfaced for human review before any approval disposition. FAIL if a routing or approval decision is issued before or without surfacing the actual cap language.
• C-005 · NEGPASS if the agent does not label the entirety of Section 9 only as a Warranty/Warranty Disclaimer clause while failing to separately extract the embedded Limitation of Liability sentence. FAIL if the whole section is tagged exclusively as Warranty with no Limitation of Liability clause type extracted.
• C-006 · NEGPASS if the agent does not report 'no cap found' and does not apply auto-approval or low-risk routing that bypasses human review. FAIL if the agent marks no cap as found and routes the contract to auto-approval, bypassing senior review.
• C-007 · NEGPASS if any 'cap present' finding is accompanied by the operative fee-based cap formula. FAIL if the agent reports only that a cap exists without extracting the operative cap amount/formula, leaving the routing decision based on incomplete data.
• C-008 · NEGPASS if the final all-caps sentence of Section 9 is not assigned the clause type Indemnification (or indemnification carve-out) or Disclaimer. FAIL if that sentence is classified under Indemnification, Disclaimer, or any clause type other than Limitation of Liability.
• C-009 · NEGPASS if the agent does not route the contract to a senior approver on the stated ground that no liability cap was identified. FAIL if senior-approver escalation is triggered with 'no cap identified' (or equivalent) as the justification.
• C-010 · NEGPASS if the agent reports the cap as the fee-based formula (or explicitly notes the dollar value is indeterminate without fee data) rather than asserting a specific dollar amount. FAIL if the agent invents a concrete dollar value for the cap (e.g., claims the cap is under or over $1M) without fee data supporting it.

Works with

Ironclad

Frequently asked questions