Workflows Workflow Builder

How this eval is graded

Pass/fail checks, each adjudicated by an LLM judge.

Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.

• C-001PASS if the system rejects the publish action and no state change to 'published' is written for this workflow version. FAIL if the workflow record is saved with a published, active, or any non-draft status.
• C-002PASS if the workflow version retains 'draft' status immediately after the rejected publish attempt. FAIL if the workflow version transitions to any state other than 'draft' (e.g., 'published', 'pending', 'active').
• C-003PASS if the error message identifies the first step by its numeric position ('Step 1') or its configured step name. FAIL if the error refers only to 'a step' or omits any step identifier.
• C-004PASS if the error message references the unbound parameter by its exact label 'Document Set'. FAIL if the error uses only a generic reference such as 'a required field' or 'an input' without naming the parameter.
• C-005PASS if the error message instructs the user to resolve the issue by either binding the parameter to a user-supplied input field or mapping it from a prior step's output. FAIL if the error states the problem without offering any remediation guidance.
• C-006PASS if no 'Run', 'Execute', or 'Schedule' action is accessible or enabled on this workflow version after the failed publish attempt. FAIL if any such action is clickable or callable on this version.
• C-007PASS if dismissing or closing the error UI element alone does not clear the blocking state and the error re-triggers on any subsequent publish attempt without the binding resolved. FAIL if the user can dismiss the error and proceed toward publication without binding 'Document Set'.
• C-008 · NEGPASS if any publish attempt with 'Document Set' unbound triggers a visible, blocking error before completing. FAIL if the builder completes publication without surfacing any error when the parameter is unbound.
• C-009 · NEGPASS if the error message specifically identifies both the step and the parameter name. FAIL if the builder issues only a generic warning such as 'some fields are incomplete' without naming Step 1 or 'Document Set'.
• C-010 · NEGPASS if any value supplied for 'Document Set' at or before publication is validated against the expected type (document collection reference or structured step-output mapping). FAIL if the builder accepts an arbitrary free-text string as a valid binding without type-checking.
• C-011 · NEGPASS if the execution harness refuses to run the workflow and does not infer a document set from ambient context when the parameter has no binding. FAIL if the harness silently selects whichever document is currently in scope and proceeds with execution.
• C-012 · NEGPASS if the validation error is raised when the user clicks 'Publish to Firm Library', before any run is attempted. FAIL if no error is shown at publish time and the error only appears when a user subsequently attempts to execute the workflow.

Works with

Legora

Frequently asked questions