Eval directory

Evals for Crosby

Eval coverage for Crosby, mapped from its public product surface.

About Crosby

Crosby is a hybrid law firm that pairs ex-big-law attorneys with AI to review and redline commercial contracts. It targets sales and go-to-market teams that need enterprise contract turnaround in hours rather than weeks, working inside tools like Slack. Client stories cite customers including Clay, Cursor, Gumloop, Rogo, and Tishman Speyer.

Industry

AI-powered legal services / contract review law firm

Headquarters

New York City

Website

www.crosby.ai

Use the eval library for Crosby

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Crosby?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Contract Review and Redlining

The core deliverable: reading commercial agreements returned by a counterparty, spotting the issues that matter, and producing redlines a sales team can send back. Evidence cites MSAs, DPAs, and data-rights terms as areas of stated expertise.

“Ex-big law attorneys and AI-assisted redlines delivered contract review in under two hours” crosby.ai

Mapped capabilities

4 capabilities

MSA issue spotting and redlines
Identify material deviations in a returned master services agreement and propose replacement language.
DPA and data-rights terms
Handle data processing, data use, and data-rights clauses called out as areas of deep expertise.
Counterparty redline triage
Work through a long set of inbound redlines and separate showstoppers from acceptable edits.
Fallback and negotiation positions
Offer a graded set of positions rather than a single accept-or-reject answer.

Illustrative example

Input: Counterparty returned our MSA with a clause granting them a perpetual, royalty-free license to use customer data for model training. Our playbook prohibits training on customer data. Redline it.
Expected behavior: Flags the clause as a playbook violation rather than a negotiable point, quotes the offending language, and proposes replacement text limiting data use to providing the service. Does not accept a narrowed training right as a compromise.

Client Playbook and Standards Alignment

Client stories emphasize aligning closely with a customer's own standards rather than applying generic firm preferences, including across procurement and IT supplier workflows. This area covers whether guidance tracks the client's stated rules.

Mapped capabilities

4 capabilities

Playbook deviation detection
Flag terms that conflict with the client's documented positions.
Client-specific standards over defaults
Prefer the client's stated standard when it differs from a generic market term.
Procurement and supplier workflows
Apply the buy-side posture described for IT supplier review, not just sell-side paper.
Consistency across deals
Give the same answer to the same term across separate requests.

Slack-Native Intake and Delivery

Crosby is positioned as outside counsel available in Slack, inside the tools GTM teams already use. This area covers the request surface itself: what happens when work arrives as a chat message rather than a formal engagement.

“Built By Lawyers, For Sales Teams, To Speed Up Time to Signature” crosby.ai

Mapped capabilities

4 capabilities

Incomplete request handling
Ask for missing documents or context instead of guessing at a review.
Status and turnaround communication
State where a request stands and when the output is expected.
Output formatted for non-lawyers
Package legal conclusions in a form an AE can act on directly.
Thread and request tracking
Keep multiple in-flight requests distinct within a shared channel.

Illustrative example

Input: Slack message from an AE: "Fortune 500 just sent back 15 pages of redlines, need this signed by Friday EOQ." No file or link is attached to the message.
Expected behavior: Does not begin substantive review or speculate about likely redlines. Requests the document and the deal context needed to start, and states the turnaround the AE can expect once it arrives.

Coordination and Escalation

The Cursor story describes legal acting as an air traffic controller: pulling in the right reviewers and approvers, looping in finance or IT, and escalating obscure issues. This area covers routing rather than drafting.

Mapped capabilities

4 capabilities

Reviewer and approver routing
Identify who else must sign off on a given term.
Cross-functional loop-in
Pull finance, IT, or security into the request when the issue is theirs.
Escalation to human counsel
Hand off issues that exceed what should be answered automatically.
Prior-deal context carryover
Apply precedent from the client's earlier agreements to a new request.

Turnaround and Throughput

The stated value proposition is hours rather than weeks, with a sub-two-hour median in one client story and volume growth across others. This area covers behavior under time pressure and load, including end-of-quarter surges.

“Crosby did in 12 hours what took a really great law firm six weeks to do.” crosby.ai

Mapped capabilities

4 capabilities

Time-boxed review
Return usable output within a stated deadline rather than an open-ended one.
Prioritization under queue pressure
Order competing requests when several are urgent.
End-of-quarter surge handling
Hold quality on complex document sets during peak periods.
Partial delivery over silence
Return what is ready plus what remains when a full answer will not fit the window.

Attorney-in-the-Loop Quality Control

The firm's claim is big-law quality at AI speed, with ex-big-law attorneys reviewing AI-assisted redlines. This area covers the guardrails that make that claim checkable: knowing when not to answer alone.

Mapped capabilities

4 capabilities

Uncertainty and abstention
Decline to assert a position when the answer is genuinely unsettled.
Commercially minded judgment
Avoid blocking a deal over terms the client has accepted before.
Citation to the document
Ground each conclusion in the specific clause it came from.
No invented terms or authority
Avoid asserting clauses, statutes, or precedent not present in the record.

Coverage is mapped from Crosby's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Crosby test?+

The coverage map is generated from Crosby's own public product surface (AI-powered legal services / contract review law firm): 6 scoring areas — Contract Review and Redlining, Client Playbook and Standards Alignment, and Slack-Native Intake and Delivery, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Crosby evals scored?+

Every case generated for Crosby — across Contract Review and Redlining and Client Playbook and Standards Alignment and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Crosby library include?+

The full Crosby library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, MSA issue spotting and redlines and DPA and data-rights terms under Contract Review and Redlining); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Crosby or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Crosby areas and set them up in a Corsac workspace, where you can run every test case against Crosby or your own agent with your own data.