Eval directory

Evals for Kea

Eval coverage for Kea, mapped from its public product surface.

About Kea

Kea AI is a voice and text AI phone agent built for restaurants that answers calls and takes orders 24/7. It connects directly to a restaurant's POS and sends orders to the kitchen display system, following the restaurant's actual menu including modifiers. Operators manage greetings, FAQs, and brand voice through the Kea AI Portal and get real-time call reporting on missed-call revenue.

Industry

restaurant voice AI phone ordering

Headquarters

Mountain View, California

Website

kea.ai

Use the eval library for Kea

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Kea?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Order Capture Accuracy

The core transaction: turning a spoken or texted order into a correct, complete cart. Covers item identification, quantity, modifier handling ('extra crispy'), and confirming the order back to the guest before submission.

Mapped capabilities

4 capabilities

Item and quantity resolution
Maps spoken item names, abbreviations, and plurals to the correct menu SKU with correct counts.
Modifier and customization handling
Applies add/remove/substitute modifiers to the right line item, including stacked and conflicting modifiers.
Order readback and confirmation
Summarizes the full cart and total back to the guest and captures explicit confirmation before submitting.
Mid-call order edits
Handles additions, removals, and corrections made after items are already in the cart.

Illustrative example

Input: Caller orders two 8-piece wings, then says: "Actually make the first one extra crispy with ranch on the side, and keep the second one plain."
Expected behavior: The agent attaches the extra-crispy and ranch-on-the-side modifiers to only the first wings line item, leaves the second with no modifiers, and reads the two items back distinctly before asking the guest to confirm.

Menu Grounding and Hallucination Control

The AI Menu Analyzer claim: the agent follows the restaurant's actual curated menu and does not invent items, modifiers, or prices. This is the highest-stakes correctness surface because errors reach the kitchen.

“Your menu is curated to your customer base, so our AI follows it. No hallucinations of 10 ketchups.” kea.ai

Mapped capabilities

4 capabilities

Off-menu request refusal
Declines items and modifiers not on the location's menu instead of accepting them.
Price and availability fidelity
Quotes only prices and availability sourced from the configured menu; defers rather than guessing.
Sold-out and 86'd item handling
Recognizes unavailable items and offers configured alternatives without fabricating substitutes.
Implausible quantity guardrails
Catches nonsensical modifier or quantity requests rather than passing them through to the ticket.

Illustrative example

Input: Caller: "Hi, can I get a large lobster mac and cheese and a Diet Coke?" The location's configured menu contains no lobster mac and cheese; Diet Coke is on the menu.
Expected behavior: The agent adds the Diet Coke, states that lobster mac and cheese is not on the menu, and either offers a configured menu alternative or asks what else the guest would like. It does not add the item, quote a price for it, or promise the kitchen can make it.

Call Handling Under Load

24/7 answering with concurrency: multiple simultaneous callers, peak-hour volume, and kitchen throttling when tickets back up. Tests availability behavior rather than conversation content.

“5 callers at once? No problem. Tickets backed up? Kea AI throttles orders.” kea.ai

Mapped capabilities

4 capabilities

Concurrent call handling
Maintains separate, uncrossed order state across simultaneous callers.
Order throttling and quote times
Adjusts promised ready times or paces orders when the kitchen is backed up.
Interruption and barge-in recovery
Handles guests talking over the agent, background noise, and partial utterances without losing cart state.
Escalation to staff
Routes to a human or takes a message when the request falls outside what the agent can complete.

Brand Voice and FAQ Configuration

Portal-controlled behavior: greetings, dynamic FAQs, and brand voice. Tests that operator configuration actually constrains the agent and that the agent stays inside its answerable scope.

“With the Kea AI Portal, you control how the AI reflects your brand.” kea.ai

Mapped capabilities

4 capabilities

Configured greeting fidelity
Uses the operator's location name and greeting text as configured.
FAQ answer grounding
Answers hours, location, and policy questions from configured FAQ content, not general knowledge.
Out-of-scope question boundary
Declines or defers questions with no configured answer instead of improvising a policy.
Tone and persona consistency
Holds the configured brand voice across a full call, including under guest frustration.

POS, KDS, and Payment Handoff

Downstream fidelity: the order the guest confirmed is the ticket the kitchen sees, and pickup/delivery/payment details survive the handoff. Includes failure behavior when an integration is unavailable.

Mapped capabilities

4 capabilities

Cart-to-ticket fidelity
Submitted POS/KDS ticket matches the confirmed cart line-for-line including modifiers.
Fulfillment mode capture
Correctly records pickup versus delivery along with required address and contact details.
Payment flow handoff
Routes the guest to Kea Pay or the configured payment path without mishandling payment data.
Integration failure behavior
Tells the guest honestly and preserves or safely aborts the order when POS submission fails.

Cross-Channel and Call Reporting

The voice/text bridge and the reporting layer operators use to see missed-call revenue and customer question trends. Tests channel parity and the honesty of derived metrics.

“On average, restaurants save $677 per month in missed order revenue.” kea.ai

Mapped capabilities

3 capabilities

Voice/text behavior parity
Same menu grounding and order rules apply whether the guest calls or texts.
Call outcome classification
Labels calls as ordered, inquiry, missed, or escalated in a way that matches the transcript.
Reporting claim accuracy
Revenue and missed-call figures are traceable to logged calls rather than estimated without basis.

Coverage is mapped from Kea's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Kea test?+

The coverage map is generated from Kea's own public product surface (restaurant voice AI phone ordering): 6 scoring areas — Order Capture Accuracy, Menu Grounding and Hallucination Control, and Call Handling Under Load, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Kea evals scored?+

Every case generated for Kea — across Order Capture Accuracy and Menu Grounding and Hallucination Control and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Kea library include?+

The full Kea library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, Item and quantity resolution and Modifier and customization handling under Order Capture Accuracy); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Kea or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Kea areas and set them up in a Corsac workspace, where you can run every test case against Kea or your own agent with your own data.