Assistant Deep Analysis And Long Form Memo Generation

How this eval is graded

Pass/fail and graded 1–5 by an LLM judge, depending on the test.

Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.

• C-001PASS if the decomposition plan contains exactly eight labeled sub-issue entries, one corresponding to each of the eight questions stated in the user prompt. FAIL if the plan contains fewer than eight entries for any reason.
• C-002 · NEGPASS if the complete decomposition plan is output as a visible, inspectable artifact before any document retrieval, vault query, or attachment read operation is initiated. FAIL if any retrieval operation begins before the plan is presented to the user.
• C-003 · NEGPASS if enforceability of the non-compete under Delaware law (issue 1) and enforceability under California law (issue 2) are represented as two distinctly labeled, separate entries in the plan. FAIL if they are merged into a single 'non-compete enforceability' or jurisdictionally undifferentiated entry.
• C-004 · NEGPASS if the likelihood of obtaining a TRO based on the employee declaration (issue 5) appears as its own labeled entry in the plan. FAIL if this issue is absent from the plan, including when the omission occurs because the issue is attachment-dependent rather than a pure legal question.
• C-005 · NEGPASS if the choice-of-law analysis for the non-solicitation clause (issue 8) appears as a distinct labeled entry separate from the independent enforceability of the non-solicitation clause (issue 3). FAIL if issue 8 is folded into issue 3 without explicit disclosure of the merger.
• C-006 · NEGPASS if issues listed toward the end of the user prompt — damages methodology (issue 6), arbitration clause (issue 7), and choice-of-law for non-solicitation (issue 8) — are all present in the plan. FAIL if one or more of these late-listed issues is absent while earlier issues are included.
• C-007PASS if the independent enforceability of the non-solicitation clause contingent on the non-compete being voided (issue 3) appears as its own labeled entry distinct from either non-compete enforceability entry. FAIL if it is absent or subsumed under issue 1 or issue 2.
• C-008PASS if the scope of the trade secrets claim under the Defend Trade Secrets Act (issue 4) appears as a distinct labeled entry. FAIL if it is absent or merged with another issue without disclosure.
• C-009PASS if the damages calculation methodology for lost client revenue (issue 6) appears as a distinct labeled entry in the plan. FAIL if it is absent or merged with another issue.
• C-010PASS if the question of whether the arbitration clause compels arbitration of this dispute (issue 7) appears as a distinct labeled entry in the plan. FAIL if it is absent or merged with another issue.
• C-011PASS if the number of plan entries is either stated explicitly as a count or can be unambiguously determined by counting visually discrete, labeled items in a structured list. FAIL if the plan is rendered as undifferentiated prose that makes counting plan entries ambiguous without inference.

Works with

Harvey

Frequently asked questions