Agent Orchestration Plan Execute Review Deliver Loop

How this eval is graded

Pass/fail and graded 1–5 by an LLM judge, depending on the test.

Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.

• C-001PASS if the plan contains a discrete, explicitly named step dedicated solely to identifying the governing law of each contract (e.g., 'Identify governing law per contract' or equivalent). FAIL if no such dedicated step appears anywhere in the plan.
• C-002PASS if the governing-law identification step is assigned a sequence index strictly lower than every jurisdiction-specific change-of-control analysis step in the plan. FAIL if any jurisdiction-specific analysis step appears at the same or a lower sequence index than the governing-law identification step.
• C-003PASS if every jurisdiction-specific change-of-control analysis step in the plan explicitly references the governing-law identification step as a named dependency or hard prerequisite in the plan's dependency graph. FAIL if any such analysis step omits or does not name this dependency.
• C-004PASS if the plan includes separate or clearly differentiated analysis steps for each relevant jurisdiction class (e.g., Delaware, Ontario, England & Wales or materially equivalent groupings). FAIL if the plan provides only a single undifferentiated change-of-control analysis step that does not distinguish between governing jurisdictions.
• C-005PASS if the governing-law identification step is explicitly scoped to all 40 contracts or the full contract set without carve-outs. FAIL if the step is scoped to a subset of contracts or does not account for the entire document population described in the task.
• C-006 · NEGPASS if no version of the plan places the governing-law identification step and any jurisdiction-specific threshold analysis step within the same concurrent execution group or parallel batch. FAIL if the plan permits governing-law identification and change-of-control analysis to run simultaneously for any contract or contract batch.
• C-007 · NEGPASS if the plan does not contain a single undifferentiated step (e.g., 'Analyze all contracts for change-of-control triggers') that appears before or merges with the governing-law identification step. FAIL if such a merged or pre-positioned step exists and would allow substantive analysis to begin before governing law is determined.
• C-008 · NEGPASS if no analysis step in the plan assumes or applies a uniform governing law to all 40 contracts prior to the completion of the governing-law identification step. FAIL if any step explicitly or implicitly treats all contracts as subject to a single jurisdiction's standards before governing-law determination is complete.
• C-009 · NEGPASS if the governing-law identification step is designated as mandatory, required, or blocking in the plan with no language suggesting it can be skipped or treated as best-effort. FAIL if the step is labeled optional, advisory, recommended, or otherwise non-blocking, or if it lacks a dependency edge that would prevent downstream steps from executing without it.

Works with

Legora

Frequently asked questions