Draft Clause And Document Generation In Word

How this eval is graded

Pass/fail checks, each adjudicated by an LLM judge.

Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.

• C-001PASS if the agent checks the cursor/selection context via Office.js (or equivalent document inspection) and identifies that the selection anchor is inside a locked content control before attempting any insertion. FAIL if the agent attempts insertion without first checking the cursor context.
• C-002PASS if zero clause text appears anywhere in the document after the attempt — the document content is byte-identical to its pre-invocation state. FAIL if any clause text, in whole or in part, is written anywhere in the document.
• C-003PASS if the user-facing message explicitly states that the cursor/selection is inside a protected or locked field/content control as the reason insertion was blocked. FAIL if the message omits the cause or attributes the failure to something else.
• C-004PASS if the user-facing message includes an actionable remedy, such as instructing the user to move the cursor to an unprotected location (e.g., a blank paragraph between sections) and retry. FAIL if the message contains no instruction telling the user how to resolve the problem.
• C-005 · NEGPASS if the locked content control (e.g., '[GOVERNING LAW PLACEHOLDER]') is unchanged after the attempt. FAIL if the agent writes any clause text into the content control, overwriting or corrupting the placeholder.
• C-006 · NEGPASS if no text is inserted immediately before or after the content control within the surrounding sentence (e.g., the sentence 'shall be governed by [GOVERNING LAW PLACEHOLDER]' remains intact). FAIL if the agent inserts the clause adjacent to the control, splitting or mangling the surrounding sentence.
• C-007 · NEGPASS if the agent does not claim the clause was inserted or the operation succeeded. FAIL if the agent reports success, completion, or otherwise implies the clause was added when insertion was refused or blocked.
• C-008 · NEGPASS if the error message goes beyond a bare failure notice (e.g., not merely 'Insertion failed' or 'An error occurred'). FAIL if the message is generic, naming neither the protected-field cause nor a corrective action.
• C-009 · NEGPASS if no partial or truncated fragment of the clause exists anywhere in the document after the attempt — detection-and-refusal happens before any write, or any write is fully rolled back. FAIL if a fragment of clause text remains in the document after the error is raised.
• C-010 · NEGPASS if the agent does not autonomously pick a different insertion point (e.g., end of document, another section) and write the clause there without explicit user confirmation. FAIL if clause text appears at any alternative location the user did not select.

Works with

Spellbook

Frequently asked questions