Workflow Agents Governance

How this eval is graded

Pass/fail checks, each adjudicated by an LLM judge.

Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.

• C-001PASS if the workflow performs a discrete MAC clause extraction step on each agreement before any branch condition is evaluated; FAIL if the routing decision is made without a prior clause extraction step.
• C-002PASS if the condition that determines routing to the regulatory memo step explicitly references language extracted from the MAC clause of the agreement under evaluation; FAIL if the routing decision is made without citing any extracted clause text.
• C-003 · NEGPASS if the branch condition is not determined solely by document filename, file path, or document-level metadata; FAIL if the workflow routes to the regulatory memo step based solely on a filename or document-level attribute rather than on extracted clause content.
• C-004 · NEGPASS if exactly one branch (either the regulatory memo step or the standard risk summary step) executes for each agreement processed in the run; FAIL if both branches execute for the same agreement.
• C-005 · NEGPASS if the final output for a given agreement contains results from only the branch that executed; FAIL if outputs from both the regulatory memo step and the standard risk summary step are combined or merged in the response for the same agreement.
• C-006PASS if the workflow correctly identifies the presence of governmental carve-out language when that language appears in the extracted MAC clause text; FAIL if the workflow fails to detect governmental carve-out language that is present in the clause.
• C-007 · NEGPASS if the regulatory memo step receives only agreements whose extracted MAC clause is confirmed to contain a governmental carve-out; FAIL if any agreement whose MAC clause lacks a governmental carve-out is routed to the regulatory memo step.
• C-008PASS if the workflow routes to the standard risk summary step when the extracted MAC clause contains no governmental carve-out language; FAIL if the regulatory memo step executes for an agreement whose MAC clause does not contain a governmental carve-out.
• C-009 · NEGPASS if the regulatory memo step is not applied to contracts that are not M&A agreements or that do not contain a MAC clause; FAIL if non-M&A contracts or agreements lacking a MAC clause are included as inputs to the regulatory memo step.

Works with

Harvey

Frequently asked questions