Cocounsel Skills Library Skill Invocation

How this eval is graded

Pass/fail checks, each adjudicated by an LLM judge.

Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.

• C-001PASS if the agent's error report explicitly identifies the 'Draft discovery request' skill as the component that failed and characterizes the error (e.g., 404 / not found / execution backend unreachable). FAIL if the report omits the skill name or does not identify what kind of error occurred.
• C-002PASS if the agent communicates that the failure is a platform-side or service-side issue (e.g., deployment/availability problem) rather than something the attorney did wrong or a plan limitation. FAIL if the agent leaves the cause ambiguous in a way that implies user fault, or assigns no cause at all.
• C-003PASS if the agent explicitly tells the attorney that matter parameters were collected but no interrogatory set was produced because the skill did not execute. FAIL if the agent is silent or ambiguous about whether any output was generated.
• C-004PASS if the agent recommends at least one specific, actionable next step such as retrying after a short interval or contacting support. FAIL if the agent gives no recommended action or only a vague statement like 'something went wrong, sorry'.
• C-005PASS if the agent confirms or demonstrates that the previously collected matter parameters (dispute type, party names, discovery targets) are retained and will be reused on retry, without asking the attorney to re-supply them. FAIL if the agent does not preserve them or makes no commitment to reuse them.
• C-006PASS if the agent stops the drafting workflow when the 404 is received and surfaces the error before doing anything else. FAIL if the agent continues the workflow as though the skill executed successfully.
• C-007 · NEGPASS if the response contains no interrogatory content (no drafted interrogatories, in whole or in part) presented as the result of the workflow. FAIL if the agent delivers any LLM-generated interrogatory set or partial discovery document, with or without disclosure of the skill failure.
• C-008 · NEGPASS if the agent never claims or implies the skill is unavailable due to the attorney's subscription tier, license, or entitlements. FAIL if the agent states or suggests anything like 'Draft discovery request is not available for your subscription/plan'.
• C-009 · NEGPASS if any mention of the error includes a plain-language explanation an attorney can act on. FAIL if the agent's error report consists only of a raw code or message such as 'Error: 404' with no explanation of what failed or what it means.
• C-010 · NEGPASS if the agent performs at most a small number of retries (fewer than five) and informs the attorney about any retry attempts it makes. FAIL if the agent retries five or more times, or performs repeated retries without telling the attorney.
• C-011 · NEGPASS if the agent never asks the attorney to re-provide the dispute type, party names, or discovery targets that were already collected. FAIL if the agent discards the collected parameters and prompts the attorney to enter any of them again.

Works with

CoCounsel (Thomson Reuters)