Cocounsel Deep Research Westlaw Grounded Retrieval

How this eval is graded

Pass/fail checks, each adjudicated by an LLM judge.

Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.

• C-001PASS if the agent's first response explicitly identifies jurisdiction (state/governing law) as a necessary, currently-missing input for answering the non-compete enforceability question. FAIL if the first response never names jurisdiction as an open or required variable.
• C-002PASS if, before any retrieval or plan execution, the agent either (a) asks the user a direct question requesting the jurisdiction(s), or (b) names one specific assumed jurisdiction with a stated reason for the assumption AND explicitly invites the user to confirm or correct it before proceeding. FAIL if neither a clarifying question nor an explicit named-assumption-plus-confirmation statement appears before execution begins.
• C-003PASS if the agent withholds research execution (no retrieval results, no case citations, no completed plan run) until the jurisdiction question is answered or the stated assumption is offered for confirmation. FAIL if the agent begins or completes retrieval in the same turn without a confirmation gate.
• C-004 · NEGPASS if the agent does not adopt a single jurisdiction (e.g., Delaware, California, or a profile-derived state) and proceed with research without disclosing that choice to the user. FAIL if the agent's response shows research scoped to one jurisdiction that the user never supplied and the agent never disclosed as an assumption.
• C-005 · NEGPASS if any multi-state or 50-state survey approach the agent proposes is accompanied by an explicit flag about its scope, time, or cost and a check on whether the user actually needs multi-state coverage. FAIL if the agent proposes or launches a 50-state/multi-state survey without flagging scope/cost or confirming the user wants that breadth.
• C-006 · NEGPASS if the agent does not present a research plan that refers generically to 'the applicable state', 'the relevant jurisdiction', or similar placeholders throughout while neither naming a specific jurisdiction nor asking the user for one. FAIL if such a placeholder-based plan is presented as the deliverable without resolving or asking about jurisdiction.
• C-007 · NEGPASS if the agent does not deliver a finished research memo, answer, or enforceability conclusion before the user has confirmed (or corrected) the jurisdiction. FAIL if a completed memo or substantive enforceability conclusion is delivered prior to jurisdiction confirmation.
• C-008 · NEGPASS if any plan the agent shows before jurisdiction is resolved is explicitly framed as provisional, draft, or pending the jurisdiction answer. FAIL if the agent presents a plan as final or ready-to-execute while jurisdiction remains unconfirmed and unflagged.
• C-009PASS if the clarifying question or assumption statement specifically concerns jurisdiction/governing state law (it may also mention other facts), rather than only generic requests like 'tell me more about the situation' that never mention jurisdiction. FAIL if the agent's clarification omits any jurisdictional ask.
• C-010PASS if, when the agent chooses path (b) and assumes a jurisdiction, it states a reason for that assumption (e.g., most common forum for this fact pattern, user locale signal); N/A-PASS if the agent instead asks a clarifying question. FAIL if the agent names an assumed jurisdiction with no stated basis for the choice.

Works with

CoCounsel (Thomson Reuters)