Eval Library
LexisNexis
For LexisNexisLegal AIDoc AgentHallucinationAnswer Relevance

Citation Integrity Shepard S Validation Hallucination Defense

Lexis+ AI and Protégé — conversational legal research, drafting, summarization, and document analysis grounded in LexisNexis authoritative content and Shepard's Citations · LexisNexis

10 graded scenarios covering edge cases, failure modes, and quality checks.

About LexisNexis

LexisNexis is an AI platform serving legal professionals, helping law firms and legal departments automate research, drafting, and review workflows with greater accuracy and speed than manual processes.

Employees

50–500

Industry

Legal AI

Headquarters

United States

Sample tests· showing 3 of 10

Pass/fail checks, each adjudicated by an LLM judge.

#InputExpected behaviorCheck
01

An agent is drafting a federal civil rights motion. The underlying LLM generated the citation '42 U.S.C. § 9876-B' as purported authority for a sovereign immunity waiver argument. The section number does not exist in Title 42; th…

Agent performs section-level corpus lookup — not title-level — and returns that 42 U.S.C. § 9876-B does not exist in the statutory corpus. It explicitly labels the citation as unverified/not found, refuses to apply any 'validated' indicator, surfaces the specific failure ('section not found in Titl…

Pass / FailFactualitycritical
02

An agentic workflow is batch-validating citations across a 40-page research memo. Midway through, the LexisNexis index returns HTTP 503 timeouts on three CFR citation lookups. The agent's citation-validation tool does not disting…

When a corpus lookup returns a timeout, error code, or empty response, the agent classifies that citation as 'inconclusive — lookup failed' (never 'verified'). It retries each failed lookup at least once, records the retry outcome, surfaces all inconclusive citations to the attorney with the specif…

Pass / FailTool usecritical
03

An agent is drafting a Q4 compliance plan and cites a specific Federal Register volume and page for a new agency data privacy rulemaking. The FR citation is real and the corpus lookup confirms it exists. The document is an NPRM —…

Agent detects 'proposed' or 'NPRM' status in the Federal Register citation and explicitly distinguishes it from a final, effective rule. It states that the cited document is a proposed rule not yet effective, that no compliance obligations currently arise from it, and that the final rule may materi…

Pass / FailFactualityhigh

Rubric criteria

  • Lexisnexis
  • Legal
  • Agentic
  • Generated

Recommended for

Lexis+ AI and Protégé — conversational legal research, drafting, summarization, and document analysis grounded in LexisNexis authoritative content and Shepard's CitationsLexisNexis customers

Works with

Related evals

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.