Eval directory

Evals for Reply.io

Eval coverage for Reply.io, mapped from its public product surface.

About Reply.io

Reply.io is a B2B sales engagement platform that combines prospect discovery, multichannel outreach sequences (email, LinkedIn, SMS, calls), and AI SDR agents in one tool. It pairs a live B2B contact database and intent signals with email deliverability tooling such as warmups and anti-spam features. Pricing is tiered by active contacts per month, with separate Sales Outreach, AI SDR, and Agency plans.

Industry

AI sales outreach and cold email automation platform

Website

reply.io

Use the eval library for Reply.io

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Reply.io?

6 scoring areas · 21 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Prospect Discovery & B2B Data

Building ICP-matched lists from Reply's live contact database, intent signals, LinkedIn sourcing, enrichment, and site-visitor identification.

“Access over 1 billion global contacts with the latest data and intent signals” reply.io

Mapped capabilities

4 capabilities

Audience discovery & ICP list building
Filtering to accounts and contacts that match a stated ICP rather than generic pulls.
Live data & intent signals
Using real-time B2B data plus hiring, LinkedIn engagement, tech-stack, and growth signals for timing.
LinkedIn / Sales Navigator prospecting
Pulling contacts with full profile data from LinkedIn and pushing them into sequences.
Contact enrichment & scoring
Turning partial contacts into full profiles and keeping pipeline records updated.

Multichannel Sequences & Outreach

Designing and running sequences that reach prospects across email, LinkedIn, SMS, calls, and adjacent channels.

“Generate leads with multichannel sequences, automate follow-ups to book more meetings” reply.io

Mapped capabilities

4 capabilities

Email sequence design & follow-ups
Multi-step cadences with automated follow-up logic.
LinkedIn automation steps
LinkedIn actions as sequence steps, including plan-gated availability.
Calls & SMS automation
Phone and SMS steps within a multichannel cadence.
WhatsApp & Zapier any-channel steps
Semi-automated WhatsApp and custom channel steps via Zapier.

Deliverability & Sending Infrastructure

Keeping outbound in the primary inbox: mailbox capacity, warmup, pre-warmed domains, and anti-spam tooling.

“A set of features to ensure your emails reach the primary inbox, not the spam folder.” reply.io

Mapped capabilities

4 capabilities

Mailbox provisioning & sending capacity
Adding mailboxes to raise daily send capacity under the Fair Usage Policy.
Email warmup
Warmup included with every mailbox and its role before ramping volume.
Pre-warmed domains
Using pre-warmed domains for new sending infrastructure.
Anti-spam & deliverability suite
Feature set aimed at primary-inbox placement instead of spam.

Illustrative example

Input: Your pricing says unlimited mailboxes and unlimited emails. Can I send 50,000 emails a day from one account starting tomorrow?
Expected behavior: Explains that unlimited emails applies to sending within active contacts and that mailboxes are subject to the Fair Usage Policy, so daily volume is not uncapped. Points to adding mailboxes to raise daily capacity and to warmup, which is included with every mailbox.

AI SDR Agents

Reply's AI SDR offering, including the Jason agent and agentic chat, handling prospecting and reply-side work.

Mapped capabilities

3 capabilities

Agent-run prospecting
Delegating list building and outreach to an AI SDR rather than manual setup.
Agentic chat control
Instructing and steering the agent conversationally.
AI reply handling & meeting booking
Agent responses to inbound replies and progression toward booked meetings.

Plans, Metering & Account Structure

How Reply meters usage and packages capability across Sales Outreach, AI SDR, and Agency plan families.

“Pull contacts directly from LinkedIn or Sales Navigator into Reply – with full profile data included.” reply.io

Mapped capabilities

4 capabilities

Active contacts metering
Unique contacts receiving one first-step email plus unlimited follow-ups per month.
Plan tiers & billing terms
Per-user versus unlimited-user tiers, monthly versus annual pricing, and the 14-day trial.
Channel add-ons & credit limits
Per-account LinkedIn and Calls/SMS add-ons, live data credits, and website visitor reveals.
Agency & multi-account setup
Agency plan structure for operators running outbound on behalf of clients.

Illustrative example

Input: I'm on a 5,000 active contacts/month plan. If I send one opener and then six follow-ups to the same prospect this month, how much of my quota does that prospect use?
Expected behavior: States the prospect counts as one active contact, because an active contact is a unique contact receiving one first-step email plus unlimited follow-ups in the month. Does not invent a per-email or per-follow-up charge.

Inbox, Replies & Reporting

Working the response side of campaigns and reading campaign performance, including mobile access.

Mapped capabilities

2 capabilities

Unified inbox & mobile access
Reviewing and responding to prospect replies, including on mobile.
Campaign performance metrics
Open and reply rate reporting used to judge sequence health.

Coverage is mapped from Reply.io's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Reply.io test?+

The coverage map is generated from Reply.io's own public product surface (AI sales outreach and cold email automation platform): 6 scoring areas — Prospect Discovery & B2B Data, Multichannel Sequences & Outreach, and Deliverability & Sending Infrastructure, and more — spanning 21 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Reply.io evals scored?+

Every case generated for Reply.io — across Prospect Discovery & B2B Data and Multichannel Sequences & Outreach and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Reply.io library include?+

The full Reply.io library is built on request. The coverage map spans 6 areas and 21 capabilities (for example, Audience discovery & ICP list building and Live data & intent signals under Prospect Discovery & B2B Data); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Reply.io or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Reply.io areas and set them up in a Corsac workspace, where you can run every test case against Reply.io or your own agent with your own data.