Eval directory

Evals for Rep.ai

Eval coverage for Rep.ai, mapped from its public product surface.

About Rep.ai

Rep.ai is a browser-based widget and platform that helps sales teams identify, engage, and convert website visitors through chat, video, scheduling, and intent scoring. It installs via a script tag or NPM package and exposes a JavaScript API for controlling widget behavior, custom visitor identities, and event listening. It also includes an AI parallel dialer for outbound calling, a limited REST API, and integrations with CRM, calendar, and revenue tools such as Salesforce, HubSpot, Zoom, Zendesk, and Gong.

Industry

AI website visitor engagement and sales outbound platform

Website

docs.rep.ai

Use the eval library for Rep.ai

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Rep.ai?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Widget Installation & Delivery

Getting the browser-based widget onto a customer site via script tag or NPM package, obtaining the client key, and keeping the widget reachable under enterprise network and browser constraints.

Mapped capabilities

4 capabilities

Script tag vs. NPM/Yarn install
Head placement of the script tag, npm install --save @repai/widget, and RepAI("init", clientKey) bootstrapping.
API key retrieval and widget settings
Where the client key appears during onboarding and in widget settings; distinguishing widget client key from REST API key.
Content-Security-Policy and network allowlisting
CSP directives, Rep.ai domains and ports for firewalls and VPNs.
Browser compatibility and notifications
Supported browsers and notification setup so reps are alerted when visitors are ready to talk.

Illustrative example

Input: We use a React build pipeline and don't want a raw script tag. How do we install the Rep.ai widget via npm and initialize it with our client key?
Expected behavior: Gives the npm/yarn install of the @repai/widget package, then shows importing RepAI and calling the init form with the client key. Does not invent extra required config or a server-side setup step.

Widget JavaScript Control API

Programmatic control of widget behavior from the host page — opening, hiding, and triggering the widget from custom UI, plus abuse controls on who can invoke it.

Mapped capabilities

4 capabilities

Widget control methods
Using the JavaScript API to show, hide, and otherwise control widget behavior.
Custom invocation from page elements
Triggering Rep.ai from buttons and other custom behavior examples.
Virtual backgrounds
Configuring virtual backgrounds for video conversations.
Security and spam prevention
Documented safeguards against abusive or automated widget engagement.

Visitor Identity & Event Integration

Attaching known identities to browser visitors and reacting to widget lifecycle events in the host application, including attribution for inbound traffic from email sequencers.

“Rep.ai's integration with Zoom adds the ability to create Zoom type events in the Scheduling tool.” docs.rep.ai

Mapped capabilities

4 capabilities

Custom visitor identities
Passing application-side identity attributes so visitors are recognized rather than anonymous.
Event listeners
Subscribing to widget events and driving host-application reactions.
Email sequencer click-through identification
Identifying contacts who arrive from Outreach/Salesloft-style sequence links.
Visitor identification settings
Toggling auto-create companies based on visitor identification.

REST API & Visitor Data Operations

The limited server-side REST API at https://api.rep.ai for automation, plus privacy and bulk-export operations over visitor data.

“Rep.ai offers a limited REST API for our customers to use in order to help automate certain actions.” docs.rep.ai

Mapped capabilities

4 capabilities

Authentication and request format
API key header auth, JSON-encoded responses, standard HTTP status codes, root URL.
Organization resources
Documented org endpoints such as listing team members.
Scrub visitor data
Deleting or redacting visitor records on request.
Nightly data dumps
Scheduled bulk export of account data and its intended consumption pattern.

Integration Catalog & Scope Boundaries

Connecting Rep.ai to scheduling, CRM, support, and revenue-intelligence systems, and being precise about what each connector does and does not sync.

“This integration does not support Gong Engage (sequencing), or syncing recordings from Gong to Rep.ai.” docs.rep.ai

Mapped capabilities

4 capabilities

Scheduling connectors
Acuity, Calendly, Chili Piper, Google Calendar, Outlook Calendar, and the Scheduler tool.
CRM and enrichment connectors
Salesforce, HubSpot, Clearbit, Shopify, and Rep.ai Influence CRM fields for Contact and Account.
Meeting and support connectors
Zoom event creation, Zendesk (including the Zendesk App), Intercom, Slack, Microsoft Teams, Zapier.
Gong recording sync and its limits
OAuth connection, one-way push of dialer recordings, no Gong Engage and no Gong-to-Rep.ai sync.

Illustrative example

Input: Once we connect Gong, will our existing Gong call recordings flow into Rep.ai, and can we run Gong Engage sequences through the integration?
Expected behavior: Answers no to both: the integration pushes recordings from Rep.ai's AI parallel dialer into Gong via an org-admin OAuth connection, and does not support Gong Engage sequencing or syncing recordings from Gong back into Rep.ai.

Outbound Dialer & Intent Scoring

The AI parallel dialer for cold outbound and the intent/analytics configuration that decides which visitors and pages matter.

Mapped capabilities

4 capabilities

Dialer getting started
Setup and first-call workflow for Rep.ai Outbound.
Custom columns (properties)
Creating and managing custom properties on dialer records.
Do Not Call handling
Marking a number as Do Not Call and the resulting dialing behavior.
Intent scoring and engagement metrics
High-intent page preferences, Spotlight, and Engage performance metric definitions.

Coverage is mapped from Rep.ai's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Rep.ai test?+

The coverage map is generated from Rep.ai's own public product surface (AI website visitor engagement and sales outbound platform): 6 scoring areas — Widget Installation & Delivery, Widget JavaScript Control API, and Visitor Identity & Event Integration, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Rep.ai evals scored?+

Every case generated for Rep.ai — across Widget Installation & Delivery and Widget JavaScript Control API and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Rep.ai library include?+

The full Rep.ai library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Script tag vs. NPM/Yarn install and API key retrieval and widget settings under Widget Installation & Delivery); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Rep.ai or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Rep.ai areas and set them up in a Corsac workspace, where you can run every test case against Rep.ai or your own agent with your own data.