Review Redline Risk Spotting And Tracked Changes Generation

How this eval is graded

Pass/fail checks, each adjudicated by an LLM judge.

Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.

• C-001PASS if the response flags the Section 12 aggregate limitation-of-liability cap ($50,000 aggregate for all claims) as a risk or adverse provision for the Client. FAIL if the $50,000 LOL cap is not flagged as a risk item.
• C-002PASS if the response characterizes Section 9 as a first-party indemnification — i.e., covering Vendor's own direct losses arising from Client's conduct rather than (or in addition to) third-party claims asserted against Vendor. FAIL if Section 9 is not identified as first-party in scope.
• C-003PASS if the response explicitly notes that Section 9 contains no 'third-party claim' qualifier (or equivalent language limiting the indemnity to claims brought by third parties) as the textual basis for the first-party reading. FAIL if the missing third-party limiter is never cited as support.
• C-004PASS if the response explicitly states that the first-party indemnification in Section 9 undermines or effectively defeats the Section 12 LOL cap by giving Vendor an unlimited direct-recovery path against Client outside the cap. FAIL if this interaction between the two sections is not explicitly articulated.
• C-005 · NEGPASS if the response never categorizes Section 9 as a standard, customary, or routine third-party indemnification clause. FAIL if Section 9 is described as 'standard indemnification — third-party claims' or any equivalent characterization implying it only covers third-party claims.
• C-006 · NEGPASS if the response cross-references Section 9 and Section 12 as interacting risk items. FAIL if the LOL cap and the indemnification clause are flagged only as separate, standalone risk items with no mention of how one affects the other.
• C-007 · NEGPASS if the response identifies the structural risk (first-party framing creating an uncapped recovery path) rather than only complaining that Section 9's triggers are broad. FAIL if the response characterizes the Section 9 risk solely as scope over-reach or breadth of triggers without identifying the LOL-bypass effect.
• C-008 · NEGPASS if any redline or recommendation for Section 9 identifies the first-party framing (lack of a third-party-claim limiter) as the defect to fix, even if it also narrows triggers. FAIL if the response recommends narrowing the (a)-(c) triggers by subject matter while never identifying the first-party framing as the structural problem.
• C-009 · NEGPASS if the response avoids asserting that the limitation-of-liability issues are 'flagged and addressed', 'resolved', or fully covered when the cap-defeating interaction has not been remediated. FAIL if the response marks the LOL section as addressed or the review as complete in a way that papers over the unresolved cap-bypass exposure.
• C-010PASS if the risk analysis is framed from the Client's position (Vendor-favorable provisions flagged as adverse to Client). FAIL if any provision is assessed from the Vendor's perspective or a Vendor-favorable provision is described as favorable to the Client.

Works with

Spellbook

Frequently asked questions