Eval directory

Evals for Chatbase

Eval coverage for Chatbase, mapped from its public product surface.

About Chatbase

Chatbase is a no-code platform for building conversational AI agents that handle customer support, sales, and product guidance across chat, email, voice, and messaging channels. Agents are trained on a company's own documents, websites, and databases, and can take actions like ticket creation, human escalation, and lead capture. It offers a JavaScript embed and REST API for developers, plus tiered pricing from a free plan up to Enterprise with SSO, audit logs, and HIPAA-eligible options.

Industry

customer support AI agent platform

Website

www.chatbase.co

Use the eval library for Chatbase

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Chatbase?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Knowledge Grounding & Source Fidelity

Whether the agent answers from the connected documents, websites, and databases it was trained on, and behaves predictably when the answer is not in its sources. This is the foundation of the platform's 'Smart & Trainable' promise and the surface most likely to produce customer-visible errors.

“Train your AI Agent with your own documents, websites, or databases for accurate, relevant responses” www.chatbase.co

Mapped capabilities

4 capabilities

Answering from connected sources
Responses reflect the ingested documents, website content, and database records rather than generic model knowledge.
Out-of-scope and unknown handling
Questions with no supporting source produce an honest non-answer or a routed escalation instead of a fabricated one.
Source freshness and retraining
Behavior after sources change, including auto-retrain on Standard and above and stale-content handling.
Guardrail and role adherence
The agent stays inside the role and guardrails defined at build time, including refusing off-topic or prohibited requests.

Actions & Tool Execution

The agent's ability to take real actions rather than only reply: creating tickets, capturing leads, escalating to humans, and invoking pre-built integrations (Slack, Stripe, Zendesk, Calendly, web search) or custom API actions. Action selection and argument correctness are the core agentic control surface.

“Publish your agent across chat, WhatsApp, email, Slack, and more with a single click.” www.chatbase.co

Mapped capabilities

4 capabilities

Correct action selection
Choosing the right action for the request, and choosing no action when a plain answer suffices.
Ticket creation with full context
Tickets carry the conversation and the details a human teammate needs to follow up.
Lead capture
Capturing and structuring lead information through the conversation without derailing the customer's task.
Custom and integration actions
Invoking pre-built or custom API actions with well-formed arguments and handling their responses.

Handoff & Escalation

How the agent transitions work to humans through the two documented paths: async ticket creation, where the AI keeps chatting while a teammate follows up by email, and live hand-off, where a human is pulled into the active chat. Getting the timing and context transfer right determines whether deflection feels helpful or abandoning.

“AI agents that meet customers at every stage of their journey, across chat, email, and voice” www.chatbase.co

Mapped capabilities

4 capabilities

Escalation triggers
Recognizing frustration, complexity, or explicit human requests as reasons to hand off rather than continue.
Async vs. live path choice
Selecting ticket-based follow-up versus pulling a human into the live conversation based on urgency and context.
Context continuity
The full conversation and relevant details survive the handoff so the human does not restart the customer.
Post-handoff conduct
What the agent says and does while a ticket is pending or a human is joining.

Illustrative example

Input: I've been charged three times for the same order and your answers aren't helping. I want to talk to an actual person about this right now.
Expected behavior: Acknowledges the duplicate charges, then initiates a handoff — either creating a ticket with the conversation attached for billing follow-up or routing to a live human agent. Does not refuse the request, loop back into self-service, or promise a human without taking a handoff action.

Multi-Channel Deployment

The same agent is published across website chat, email, voice and telephony, WhatsApp and Meta apps, Slack, Shopify, and Zendesk. Each channel has different formatting, latency, and interaction constraints, so consistency of substance alongside adaptation of form is the thing to measure.

“Resolve complex support queries accurately across live chat, email, phone, Slack, and more.” www.chatbase.co

Mapped capabilities

4 capabilities

Cross-channel answer consistency
The same question yields substantively the same answer regardless of channel.
Channel-appropriate formatting
Output adapts to the medium, including plain-spoken phrasing for voice and telephony versus richer web chat.
Email and asynchronous threads
Handling longer-form, delayed, multi-turn exchanges where the customer is not present in real time.
Attachments and rich input
Behavior when customers send attachments on plans where that is supported.

Plan, Pricing & Entitlement Accuracy

Chatbase's own sales agent answers plan and pricing questions, and every plan gates a specific feature set, seat count, and message-credit allowance. Wrong entitlement answers create refund disputes and failed upgrades, making this a high-consequence factual surface with a fully documented ground truth.

Mapped capabilities

4 capabilities

Plan feature boundaries
Correctly attributing features such as API access, voice, helpdesk, SSO, and audit logs to the plans that include them.
Price and billing math
Monthly versus annual figures, the 20% yearly discount, seat counts, and add-on pricing stated accurately.
Credit limits and overage
Message-credit allowances, the auto-recharge add-on, and what happens when credits run out.
Upgrade and trial guidance
Recommending the plan that actually unlocks the capability a customer asked about, including trial terms.

Illustrative example

Input: I'm on the Hobby plan and I want to call your REST API from our backend. Can I do that on my current plan, or do I need to upgrade?
Expected behavior: States that API access is not included in Hobby and begins at the Standard plan, then points the user to Standard as the upgrade that unlocks it. Does not claim Hobby includes API access or invent an add-on that grants it.

Developer Integration Surface

The JavaScript embed script and REST API v2 are the programmatic entry points, covering identity verification, event listeners, streaming over SSE, structured errors, cursor-based pagination, agent management, and webhooks. Failures here are silent to end users but block integrations outright.

Mapped capabilities

4 capabilities

Embed script setup and events
Correct one-tag installation, event listener wiring, and custom initial-message configuration.
Identity verification
Securing sessions for authenticated apps and associating conversations with the right contact.
API v2 semantics
Structured error handling, SSE streaming, and cursor-based pagination used correctly in guidance and examples.
Webhooks and programmatic management
Real-time chat-event notifications and creating, updating, or configuring agents through the API.

Coverage is mapped from Chatbase's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Chatbase test?+

The coverage map is generated from Chatbase's own public product surface (customer support AI agent platform): 6 scoring areas — Knowledge Grounding & Source Fidelity, Actions & Tool Execution, and Handoff & Escalation, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Chatbase evals scored?+

Every case generated for Chatbase — across Knowledge Grounding & Source Fidelity and Actions & Tool Execution and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Chatbase library include?+

The full Chatbase library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Answering from connected sources and Out-of-scope and unknown handling under Knowledge Grounding & Source Fidelity); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Chatbase or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Chatbase areas and set them up in a Corsac workspace, where you can run every test case against Chatbase or your own agent with your own data.