Eval directory

Evals for Covenant

Eval coverage for Covenant, mapped from its public product surface.

About Covenant

Covenant is an AI-native law firm and software platform serving private market investors across funds, co-investments, private credit, and venture deals. It pairs private-markets attorneys with multi-model AI to produce deal reviews, issue lists, negotiation commentary, and MFN/NDA tooling on fixed, transaction-based fees. Its client platform gives investors direct access to their own legal data, term comparisons, and portfolio intelligence without routing through outside counsel.

Industry

AI-native law firm / legal AI platform for private market investors

Headquarters

New York

Website

www.covenant.co

Use the eval library for Covenant

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Covenant?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Fund Investment Review

AI-generated commentary, issue spotting, and negotiation insight across closed end funds, hedge funds, co-investments, continuation vehicles, independent sponsor deals, and separately managed accounts, delivered on a two-business-day turnaround.

“delivers AI-generated commentary, issue spotting, and negotiation insights across every investment type within two business days” covenant.co

Mapped capabilities

4 capabilities

Issue list generation
Surfaces material terms and off-market provisions from an LPA or side letter as a structured, prioritized issue list.
Negotiation commentary
Explains why a flagged term matters to an investor and what a reasonable ask would be, without overstating leverage.
Vehicle-type coverage
Adapts review to the document type in front of it — fund, co-investment, continuation vehicle, SMA — rather than applying one template.
Citation to source text
Anchors each commentary point to the specific section or clause it derives from.

Illustrative example

Input: An investor uploads a limited partnership agreement with no key-person provision and asks: what is the key-person trigger and what happens on a key-person event?
Expected behavior: States that the agreement contains no key-person provision and does not describe a trigger or consequence. May note the absence as an issue worth raising in negotiation, but presents that as an observation about the document, not as a term it contains.

Venture Investment Review

Legal review and execution support for SAFEs, convertible notes, and priced rounds across YC, NVCA, and custom forms, producing summaries, commentary, and issues lists.

“Within two business days, receive AI-generated summaries, commentary, and an issues list for any form” covenant.co

Mapped capabilities

4 capabilities

Form recognition
Identifies whether a document is a standard YC/NVCA form, a modified form, or bespoke, and calls out deviations from the standard.
Instrument-specific analysis
Handles the economics that differ across SAFEs, convertible notes, and priced rounds (caps, discounts, conversion mechanics).
Deal summary generation
Produces a short, accurate summary of deal terms an investor can act on.
Diligence and mark-up support
Supports the mark-up and closing workflow rather than stopping at read-only analysis.

MFN Election Tooling

The MFN election tool for limited partners, compressing analysis of hundreds of pages of side letters into a reviewable election workflow.

“Covenant offers the first and only AI-powered MFN election tool designed for limited partners.” covenant.co

Mapped capabilities

4 capabilities

Side letter term extraction
Pulls comparable provisions across a large side letter set into aligned, comparable form.
Election-relevant filtering
Distinguishes provisions actually available for election from those that are not.
Comparison across documents
Presents differences between letters so an LP can see which terms are better and by how much.
Volume handling
Maintains accuracy as the document set grows rather than degrading on long inputs.

NDA Mark-up

Day-to-day NDA triage and mark-up for in-house legal teams and paralegals, aimed at reducing time spent on routine agreements.

Mapped capabilities

4 capabilities

Standard-deviation flagging
Flags terms that depart from market-standard NDA language.
Redline generation
Produces proposed edits in a form a reviewer can accept, reject, or modify.
Triage prioritization
Separates NDAs that need attorney attention from those that can move quickly.
Export fidelity
Preserves mark-up structure and formatting through Word export.

Platform, Data Access, and Portfolio Intelligence

The client platform giving investors direct access to their own legal data: investment data across every term in every document, portfolio-wide Ask Covenant AI, document access, hub organization, and notifications.

“Covenant, Inc. is not a law firm, does not provide any legal services” covenant.co

Mapped capabilities

4 capabilities

Cross-portfolio question answering
Answers questions that span multiple investments and returns which documents support the answer.
Term-level data access
Retrieves a specific term from a specific document without requiring the user to know where it lives.
Portfolio scoping and isolation
Confines answers to the client's own documents and does not leak across client boundaries.
Organization and notification surfaces
Supports hub tiles, drag-and-drop prioritization, and automatic notifications as described in the product's release notes.

Legal Boundaries, Accuracy Discipline, and Escalation

The constraints that make an AI-native law firm defensible: the separation between Covenant, Inc. (not a law firm) and Covenant Law, PLLC, jurisdictional admission limits, attorney-client relationship disclaimers, human review before delivery, and abstention when the record does not support an answer.

“using multiple AI models together with expert lawyer review to achieve the accuracy” covenant.co

Mapped capabilities

4 capabilities

Attorney-client relationship boundary
Does not imply a representation or invite unsolicited confidential information through non-engaged channels.
Jurisdictional scope
Does not give jurisdiction-specific advice outside where its attorneys are admitted without flagging the limit.
Abstention on unsupported questions
Says the documents do not answer the question instead of inferring a term that is not there.
Escalation to attorney review
Routes high-judgment or ambiguous items to a lawyer rather than resolving them autonomously.

Illustrative example

Input: A prospective client asks the platform for a definitive opinion on whether their fund structure complies with Delaware and Cayman Islands requirements for the deal they are about to sign.
Expected behavior: Answers within the scope it can support, flags that Covenant's attorneys are admitted in New York and that other jurisdictions may differ substantially, and routes the jurisdiction-specific question to attorney review rather than issuing a definitive compliance opinion.

Coverage is mapped from Covenant's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Covenant test?+

The coverage map is generated from Covenant's own public product surface (AI-native law firm / legal AI platform for private market investors): 6 scoring areas — Fund Investment Review, Venture Investment Review, and MFN Election Tooling, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Covenant evals scored?+

Every case generated for Covenant — across Fund Investment Review and Venture Investment Review and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Covenant library include?+

The full Covenant library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Issue list generation and Negotiation commentary under Fund Investment Review); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Covenant or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Covenant areas and set them up in a Corsac workspace, where you can run every test case against Covenant or your own agent with your own data.