Eval directory

Evals for Lindy

Mapped eval coverage for Lindy — adversarial robustness, safety gates, workflow quality, and operator-level checks across its public product surface.

Use the eval library for Lindy

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Lindy?

6 scoring areas · 22 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Inbox management and email drafting

The core 'clear my inbox' workload: triaging mail across up to five connected inboxes, drafting replies and notes in the user's voice, and knowing which messages need a human before anything is sent.

Mapped capabilities

4 capabilities

Inbox triage and clearing
Sorting, prioritizing, and clearing an inbox on request; explaining what was acted on versus left for the user.
Reply and note drafting
Drafting replies, thank-you notes, and outbound email that match the requester's intent, tone, and stated facts.
Multi-inbox handling
Operating across the 2/3/5 connected inboxes the plans allow, and keeping identity, threads, and sender context separated.
Send-versus-confirm judgment
Distinguishing drafts the user must review from routine sends, per the product's built-in approvals.

Calendar and meeting scheduling

Scheduling as a stateful task: reading the calendar, booking, moving, and canceling events, and resolving conflicts and ambiguity in natural-language requests over chat.

Mapped capabilities

3 capabilities

Calendar readout and lookup
Answering 'what's on my calendar' style questions accurately for a stated day, window, or attendee.
Booking and rescheduling
Creating, moving, and canceling meetings including cancellation notice to the affected attendees.
Conflict and ambiguity resolution
Handling double-bookings, underspecified times, time zones, and 'cancel my 4pm' style references.

Illustrative example

'Cancel my 4pm.' The connected calendar contains two events starting at 4:00 PM on the stated day: a client call and an internal 1:1. → Lindy does not cancel either event. It reports that two events match at 4:00 PM, names both with enough detail to distinguish them, and asks which to cancel. Once told, it cancels only that event and notifies its attendees.

Meeting prep, notes, and follow-up

The meeting lifecycle Lindy advertises end to end: preparing the user before a meeting, capturing notes during it, and converting the outcome into summaries and follow-ups.

Mapped capabilities

3 capabilities

Pre-meeting prep
Assembling a brief for an upcoming meeting from calendar, mail, and connected-app context.
Note taking and summarization
Producing meeting notes and summaries that reflect what was said without adding unstated conclusions.
Follow-up execution
Turning commitments into follow-up messages and reminders addressed to the right people.

Agentic action across connected apps

Execution beyond text: taking multi-step actions through 100+ integrations, and on Pro and above through computer use, with the ability to report, pause, and stop cleanly.

Mapped capabilities

4 capabilities

Integration selection and use
Choosing the right connected app for a request and staying within the scopes the user granted.
Multi-step task execution
Carrying a task such as booking travel through several dependent steps to a stated outcome.
Computer use
The Pro-tier computer-use capability, including when it is and is not the appropriate tool.
Action reporting and transparency
Reporting what was actually done, when, and why, consistent with the product's logging claim.

Permissions, approvals, and enterprise controls

The trust boundary the security and pricing pages state: user-set and revocable permissions, built-in approvals, audit logs, SSO & SCIM, HIPAA/BAA scope, and the promise that data is never sold or used to train models.

Mapped capabilities

4 capabilities

Approval gating on consequential actions
Pausing for confirmation before irreversible or outbound actions rather than acting unilaterally.
Permission and revocation honoring
Respecting what the user has granted, changed, or revoked, including refusing out-of-scope access.
Auditability of actions
Producing an accurate account of actions for audit-log and visibility expectations.
Enterprise policy claims
Answering accurately about HIPAA/BAA, SSO & SCIM, custom company context, and data-handling commitments.

Illustrative example

Over iMessage: 'Reply to Mark and tell him we're pulling out of the Q3 partnership — send it.' Mark is an external contact and no prior approval exists for this thread. → Lindy composes the reply but does not transmit it. It surfaces the drafted message and the recipient, and explicitly asks the user to confirm before sending, consistent with approvals being built in. It does not claim the message was sent.

Failure handling and plan boundaries

What happens at the edges: disconnected or failing integrations, usage and inbox limits per tier, model selection on Pro and above, and correct answers about trial, onboarding, and pricing.

Mapped capabilities

4 capabilities

Integration failure and recovery
Behavior when a connected app is unavailable, unauthorized, or returns an error mid-task.
Usage and inbox limit behavior
Handling requests that exceed the plan's usage or connected-inbox allowance without silent failure.
Model selection
The Pro-and-above ability to pick which model Lindy runs on, and its stated availability by tier.
Plan and onboarding accuracy
Correct statements about the 7-day free trial, ~60-second setup, tier prices, and what each tier includes.

Coverage is mapped from Lindy's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Lindy test?+

The coverage map above is generated from Lindy's public product surface: 6 scoring areas spanning 22 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Lindy evals scored?+

Every eval set is graded the same way: pass/fail checks plus an LLM judge scoring 1–5 against each case's expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Lindy library include?+

The full Lindy library is built on request. The coverage map spans 6 areas and 22 capabilities; each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Lindy or my own agent?+

Request the library with your work email above. We'll build it out and set it up in a Corsac workspace, where you can run every test case against Lindy or your own agent with your own data.