Legal Research Q A Contextual Research With Jurisdictional Coverage

How this eval is graded

Pass/fail and graded 1–5 by an LLM judge, depending on the test.

Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.

• C-001PASS if: the response explicitly acknowledges that 'the Northern District' is ambiguous (multiple federal districts bear that name) OR explicitly names one specific assumed court, and this disclosure appears BEFORE any substantive local-rule content. FAIL if: substantive local-rule content appears before any jurisdiction-identification disclosure, or no disclosure is made at all.
• C-002PASS if: the response either (a) asks the user/agent to confirm which Northern District court is intended before answering substantively, or (b) names the assumed court explicitly in the first sentence of the response and invites confirmation or correction. FAIL if: the response neither asks for clarification nor both names the assumed court in the first sentence and invites correction.
• C-003PASS if: when the response proceeds under an assumed court rather than asking for clarification, it states or implies a contextual basis for the assumption (e.g., prior conversation cues, user-provided facts), or no assumption is made because clarification was requested. FAIL if: the response proceeds under an assumed court while citing no contextual basis whatsoever for choosing that court over other Northern Districts.
• C-004 · NEGPASS if: the response does NOT present local rules of a single specific Northern District court (e.g., N.D. Cal., N.D. Ill., N.D. Tex.) as the answer without flagging the multi-district ambiguity or naming the assumption. FAIL if: local rules for one specific court are returned as if the intended court were obvious, with no ambiguity flag or named assumption.
• C-005 · NEGPASS if: any content that mixes a specific district's local rules with the Federal Rules of Civil Procedure clearly labels which provisions are local rules and which are FRCP, or no such mixing occurs. FAIL if: the response blends a Northern District court's local rules with FRCP provisions without distinguishing which is which.
• C-006 · NEGPASS if: the response does NOT present generally applicable FRCP electronic-service provisions (e.g., Rule 5 content) under a heading or framing claiming they are 'Northern District' local rules. FAIL if: generic FRCP content is labeled or framed as the local rules of a Northern District court.
• C-007 · NEGPASS if: any jurisdiction assumption or ambiguity flag appears in the primary response text where a downstream agent or attorney reading the answer would see it (e.g., the opening sentence or a prominent leading statement). FAIL if: the assumption or ambiguity flag appears only in a footer, disclaimer block at the end, citation metadata, or other field separate from the main answer body.
• C-008PASS if: when flagging ambiguity, the response makes clear that more than one federal 'Northern District' court exists (e.g., by saying multiple districts share the name or by listing examples), rather than using vague hedging like 'rules may vary'. FAIL if: the response uses only generic hedging without conveying that 'Northern District' maps to multiple distinct courts.
• C-009 · NEGPASS if: the response does NOT assert specific local-rule numbers, deadlines, or e-service requirements as authoritative for a court whose identity has not been confirmed or explicitly assumed in the response. FAIL if: specific rule citations or requirements are asserted as the governing local rules while the court identity remains unconfirmed and unstated.
• C-010PASS if: the response, in addition to handling the ambiguity, offers a constructive path forward (e.g., asks which state/court, lists candidate Northern Districts to choose from, or offers to proceed once confirmed). FAIL if: the response refuses or stalls without giving the agent any actionable way to resolve which court is intended.

Works with

Paxton AI

Frequently asked questions