Vlex Ask A Research Question Grounded Legal Q A

How this eval is graded

Pass/fail and graded 1–5 by an LLM judge, depending on the test.

Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.

• C-001PASS if the memo contains a clearly labeled, structurally distinct section analyzing non-compete enforceability under Texas law that cites Texas Bus. & Com. Code § 15.50 and/or Texas appellate case law on reasonableness of time and geographic limitations. FAIL if no dedicated enforceability section exists or it contains no Texas statutory or appellate authority.
• C-002PASS if the memo contains a separately headed, structurally distinct section addressing damages exposure arising from the cease-and-desist letters already sent (e.g., tortious interference with prospective business relations, attorneys' fees under applicable Texas statutes). FAIL if no such dedicated section exists in the memo.
• C-003PASS if the damages/C&D section is supported by at least one citation that is independent of (not merely repeated from) the enforceability section's authorities, OR the section explicitly states that corpus coverage is insufficient to cite authority. FAIL if the damages/C&D section makes substantive claims with no citations of its own and no coverage-gap statement.
• C-004PASS if, given the retriever's minimal damages signal, the output either delivers substantive damages analysis or includes a user-visible, explicit notice that retrieval/corpus coverage for the damages/C&D issue was insufficient and the issue remains unresolved. FAIL if thin coverage results in neither substantive analysis nor an explicit gap notice.
• C-005PASS if the downstream agent marks the research task complete only when both the enforceability section and the damages/C&D section (or an explicit gap notice for the latter) are structurally present in the captured memo. FAIL if the task is marked complete while one of the two issues is structurally absent without a gap notice.
• C-006 · NEGPASS if the damages and cease-and-desist exposure issue appears in the output either as a substantive section or as an explicit omission/coverage-gap flag. FAIL if the damages/C&D issue is entirely absent from the memo with no acknowledgment that it was omitted, regardless of how thorough the enforceability analysis is.
• C-007 · NEGPASS if damages/C&D exposure is treated under its own dedicated heading with its own analysis. FAIL if damages is mentioned only in passing inside the enforceability section — a single merged section with no dedicated damages heading and no independent damages citations.
• C-008 · NEGPASS if, when the initial retrieval yields strong enforceability material but minimal damages material, the agent triggers a re-query, follow-up retrieval, or escalation for the damages issue before (or instead of) marking the task complete. FAIL if the agent marks the research complete based solely on a thorough Issue 1 memo and never attempts any re-query or escalation for Issue 2.
• C-009 · NEGPASS if any limitation language in the memo specifically identifies that the damages/C&D issue was not (or only partially) addressed. FAIL if the memo's only caveat is a generic boilerplate disclaimer (e.g., 'consult an attorney') that does not name the unaddressed damages/C&D issue.
• C-010PASS if the system's output reflects decomposition of the compound query into two distinct legal issues (non-compete enforceability; damages exposure from prior C&D letters), evidenced by separate treatment of each in the memo structure. FAIL if the output treats the request as a single undifferentiated question.

Works with

vLex (Vincent AI)