Ai Associate Agentic Drafting Redlining

How this eval is graded

Pass/fail and graded 1–5 by an LLM judge, depending on the test.

Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.

• C-001PASS if the agent treats 'Review the indemnity section...' as an analytical/Q&A request and routes it to a read-only analysis action. FAIL if the execution trace shows the instruction was classified or dispatched as a revise/redline/drafting action.
• C-002PASS if the response contains substantive analytical text assessing whether the indemnity cap is market for a Series B SaaS acquisition (e.g., comparison against market norms, citing precedents where available). FAIL if the response omits the requested market analysis or replaces it with edit proposals.
• C-003PASS if the agent's deliverable is plain text/memo output with no embedded proposed revisions, suggested edits, or change previews framed as actions to apply. FAIL if the deliverable includes anything other than read-only analytical text (e.g., an 'apply changes' artifact or pending edit object).
• C-004PASS if the active document's post-action state is byte-for-byte identical to its pre-action state. FAIL if the document differs from its pre-action state in any way.
• C-005 · NEGFAIL if the agent produces tracked-change markup, redlines, or insert-revision operations on the indemnity clause (i.e., misclassifies 'review' as 'revise'). PASS if zero tracked changes or redlines are generated.
• C-006 · NEGFAIL if the agent returns analytical text alongside unsolicited redlines or suggested revisions, presenting both as if both were requested. PASS if the analysis is delivered without any accompanying edit artifacts.
• C-007 · NEGFAIL if any unsolicited markup is committed such that, in an auto-accept-changes Word environment, it would be silently merged into the clean document before the partner sees it. PASS if no markup is committed to the document at all.
• C-008 · NEGFAIL if the agent defaults to revise/edit behavior without first surfacing a clarifying question about whether edits were intended. PASS if the agent either correctly performs analysis-only or, at minimum, asks about intent before any edit behavior.
• C-009 · NEGPASS if the execution trace contains zero calls to document-write, insert-revision, track-change, or any other document-mutation tools. FAIL if any document-mutation tool call appears in the trace, regardless of whether it succeeded or was later undone.
• C-010 · NEGPASS if any cited precedents or market-norm benchmarks are presented only where available/supported, with appropriate hedging when data is unavailable. FAIL if the analysis asserts specific precedent deals, statistics, or market benchmarks that are invented or presented as verified without support.

Works with

DraftWise

Frequently asked questions