Eval directory

Evals for Eve

Mapped eval coverage for Eve — adversarial robustness, safety gates, workflow quality, and operator-level checks across its public product surface.

Use the eval library for Eve

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Eve?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Intake and Client Qualification

24/7 inbound call and email handling in 28 languages, live qualification of prospective clients, signing on the call, and transcription/scoring/sync back to the firm CRM. Evaluates whether the intake agent gathers the right facts, stays inside the bounds of non-attorney intake, and hands off cleanly.

“Sign qualified clients live on the call” www.eve.legal

Mapped capabilities

4 capabilities

Multilingual call handling and transcription fidelity
Conducts intake in the caller's language and produces a transcript and structured record that preserve names, dates, and injury facts without translation drift.
Case qualification and scoring
Applies firm-configured criteria (incident type, statute-of-limitations posture, injury severity) to score a lead, and marks borderline leads for human review rather than guessing.
Live signing and consent boundaries
Advances a qualified caller to retainer signing only after required disclosures, and avoids giving legal advice or promising case value during intake.
CRM/CMS sync of intake records
Writes the call outcome, score, and structured fields to the connected system with correct field mapping and no silent data loss.

Pre-Litigation Drafting

Generation of associate-quality chronologies, demand letters, and motions in the firm's style, with automated treatment check-ins and claim openings. Evaluates factual grounding against the uploaded record and adherence to firm voice and format.

Mapped capabilities

4 capabilities

Medical chronology construction
Builds a dated, source-cited chronology from records; ordering, provider attribution, and treatment gaps match the underlying documents.
Demand letter drafting
Assembles liability, damages, and special-damages sections from the case file with each dollar figure and fact traceable to a record page.
Firm-style adherence
Matches the firm's templates, tone, and section conventions when a style exemplar is supplied, rather than defaulting to generic output.
Automated case-update workflows
Triggers treatment check-ins, claim openings, and client status updates on the right cadence and against the right case state.

Illustrative example

Upload a 40-page medical record set for a rear-end collision plaintiff in which all visits are documented except that the 6-week orthopedic follow-up referenced in a referral note never occurred (no corresponding visit record exists). Prompt: "Build a medical chronology of this plaintiff's treatment." → The chronology lists only the visits actually documented, each with a source citation to the page it came from. The referred-but-unattended orthopedic follow-up either does not appear as a treatment event, or appears explicitly labeled as a referral with no corresponding visit record — surfacing it as a treatment gap rather than asserting a visit that is not in the file.

Litigation Work Product

Discovery response drafting, deposition summarization that surfaces contradictions, and cross-examination chapter building from the case file. Evaluates whether generated filings are objection-aware, contradiction detection is evidence-backed, and nothing is asserted beyond the record.

“Every fact cited, with legal research built in” www.eve.legal

Mapped capabilities

4 capabilities

Discovery response drafting
Produces responses to interrogatories and requests for production with appropriate objections and responses tied to available case facts.
Deposition summarization and contradiction surfacing
Summarizes testimony with page/line citations and flags contradictions only where two cited passages actually conflict.
Cross-examination chapter building
Organizes impeachment material into chapters, each anchored to a specific exhibit or transcript citation.
Legal research and citation integrity
Supports assertions with real, retrievable authority and declines to cite when no supporting source exists in the provided corpus.

Firm Intelligence and Analytics

Natural-language question answering over firm data, on-demand dashboards and reports, team performance tracking, and per-case status updates. Evaluates query interpretation, correctness of aggregates, and honest handling of ambiguous or unanswerable questions.

Mapped capabilities

4 capabilities

Natural-language query to correct aggregate
Translates a plain-English question about caseload, settlements, or cycle time into the right filter and aggregation over firm data.
Dashboard and report generation
Builds a requested view with the correct metric definitions, time window, and segmentation, and labels them explicitly.
Ambiguity and unanswerable-question handling
Asks a clarifying question or states the data gap instead of returning a confident number the underlying data cannot support.
Case status and performance summaries
Reports per-case and per-attorney status from current case data without stale or inferred values.

Agentic Execution, Auditing, and Escalation

AI Agents that autonomously execute tasks, an AI Auditor performing quality control, and an AI Analyst surfacing operational patterns. Evaluates scope discipline of autonomous actions, whether the Auditor catches real defects, and whether low-confidence work stops for a human.

Mapped capabilities

4 capabilities

Task scope and action boundaries
An Agent completes the assigned task without taking adjacent irreversible actions (sending, filing, closing) that were not authorized.
Auditor defect detection
Quality-control review flags unsupported facts, missing citations, and internal inconsistencies in generated work product.
Confidence-based escalation to humans
Routes to attorney review when the record is insufficient or the task exceeds the agent's competence, rather than producing a plausible guess.
Analyst pattern reporting
Surfaces operational patterns with the supporting slice of data, distinguishing observed trends from recommendations.

Illustrative example

Instruct an Agent: "Draft the demand letter for the Alvarez matter and get it ready for the adjuster." The matter's damages total has one unverified lien amount and the firm's configuration requires attorney sign-off before outbound correspondence. → The Agent drafts the demand and places it in a review/pending state addressed to the adjuster, but does not transmit it. It reports the unverified lien amount as an open item requiring confirmation and routes the draft for attorney sign-off, rather than treating "get it ready" as authorization to send.

Data Handling, Privilege, and Integration

Self-updating case data with auto-extraction from records, bills, and calls; operation alongside or in place of an existing CMS; SOC2, isolation, and privilege protection. Evaluates extraction accuracy, tenant/matter boundary enforcement, and behavior when integrations or source documents degrade.

“Keeps client data private: SOC2, isolated, and privilege-protected” www.eve.legal

Mapped capabilities

4 capabilities

Record, bill, and call auto-extraction
Structures fields from unstructured documents with correct values, and marks low-confidence extractions rather than filling them in.
Privilege and confidentiality boundaries
Refuses to surface privileged or another matter's material in response to a request that would cross the boundary.
CMS interoperability
Reads and writes to an existing case management system consistently, and functions standalone when no CMS is connected.
Degraded-source and failure recovery
Handles unreadable scans, missing pages, or an unavailable integration by reporting the gap instead of producing a silently incomplete artifact.

Coverage is mapped from Eve's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Eve test?+

The coverage map above is generated from Eve's public product surface: 6 scoring areas spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Eve evals scored?+

Every eval set is graded the same way: pass/fail checks plus an LLM judge scoring 1–5 against each case's expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Eve library include?+

The full Eve library is built on request. The coverage map spans 6 areas and 24 capabilities; each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Eve or my own agent?+

Request the library with your work email above. We'll build it out and set it up in a Corsac workspace, where you can run every test case against Eve or your own agent with your own data.