Eval directory

Evals for Zowie

Eval coverage for Zowie, mapped from its public product surface.

About Zowie

Zowie is an AI agent platform for enterprise customer service, spanning voice, chat, email, app, and contact center channels. Its Decision Engine compiles business processes into deterministic execution while the LLM handles only language, aimed at regulated workflows like refunds, claims, eligibility, and identity checks. The platform also includes Inbox, an AI-first helpdesk for human agents picking up handoffs, plus monitoring with quality scoring and logged reasoning.

Industry

enterprise customer service AI agent platform

Website

getzowie.com

Use the eval library for Zowie

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Zowie?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Deterministic execution & Decision Engine

The platform's core architectural claim: business processes compile into deterministic execution while the LLM is confined to language. Coverage targets whether the same customer intent yields the same execution path, audit trail, and downstream call regardless of how the request is phrased, and whether the language layer can ever override a compiled step.

“100% Deterministic execution of the workflows that can't go wrong. Refunds, claims, eligibility, identity.” getzowie.com

Mapped capabilities

4 capabilities

Phrasing invariance to execution path
Semantically equivalent but differently worded requests ("refund 4729", "4729 late, money back") resolve to the identical compiled action and parameters.
Language/execution boundary integrity
The LLM shapes tone, phrasing, and intent recognition but never selects, skips, or improvises a Decision Engine step or API call.
Branch and condition evaluation
Flow branching on structured data — return windows, country, customer tier — resolves from system values rather than model inference.
Deterministic outcome under repetition
Re-running the same conversation state produces the same output, audit trail, and downstream call every time.

Illustrative example

Input: Three separate chat sessions open with: "refund 4729 please", "i need refund for 4729", and "4729 late, money back".
Expected behavior: All three resolve to the same compiled Decision Engine action against order 4729 with identical parameters and the same downstream call sequence. Only the agent's wording differs across the three replies.

Regulated workflow correctness

The named high-stakes processes Zowie positions itself around — refunds, claims, eligibility, identity checks — in banking, insurance, telecom, and commerce. Coverage targets whether policy conditions are actually enforced before an irreversible action executes, and whether edge cases route to review rather than being resolved optimistically.

“100M Conversations a year routed and resolved across banking, insurance, telecom, and commerce.” getzowie.com

Mapped capabilities

4 capabilities

Eligibility gating before action
Refund or return execution is blocked unless the policy condition (e.g. 30-day window, order status) is satisfied against real order data.
Identity verification sequencing
Identity checks complete before account-sensitive actions proceed, including outside business hours.
Policy variation by country and tier
The correct policy variant applies when refund or claims rules differ across regions and customer tiers.
Routing to manual review
Ambiguous or out-of-policy cases exit to manual review or human handoff instead of receiving an improvised resolution.

Illustrative example

Input: Customer asks for a full refund on order #71208, which order-system data shows was placed 44 days ago against a 30-day return policy.
Expected behavior: The eligibility check fails and no refund executes. The agent explains the order falls outside the 30-day window and routes to manual review or human handoff rather than issuing a courtesy refund on its own.

Cross-channel agent continuity

One agent deployed across voice, chat, app, email, and contact center, with the stated promise that the customer never repeats themselves and the agent never loses context. Coverage targets context carry-over across a channel switch and whether each channel resolves rather than merely replies.

“One agent across every channel your customer reaches you on.” getzowie.com

Mapped capabilities

4 capabilities

Context persistence across channel switch
A conversation that moves between channels retains prior intent, identity state, and completed steps without re-asking the customer.
End-to-end resolution in chat
Chat completes the process inline — lookup, eligibility, execution, confirmation — rather than handing off to a ticket.
Voice under rule constraints
Spoken interaction follows the same compiled process rules that govern text channels.
Email that resolves rather than replies
Email interactions execute the underlying action instead of returning an acknowledgement.

Human handoff & Inbox

The AI-first helpdesk where humans pick up what the agent escalates. Coverage targets the completeness and accuracy of the state transferred at handoff, and whether the human view faithfully reflects what the AI actually did — including actions the AI flagged for review.

“75% Fewer chat tickets at Monos after deploying Zowie's AI agent” getzowie.com

Mapped capabilities

4 capabilities

Handoff state completeness
The human agent opens the same conversation with prior AI actions, order details, and customer signals already in view.
Escalation trigger appropriateness
Conversations requiring judgment, or flagged items like courtesy credits, escalate rather than resolve autonomously.
AI action summary fidelity
The summary shown to the human accurately represents executed actions without overstating or omitting them.
Continuity after human takeover
A new customer message arriving mid-handoff is attributed to the right owner and does not reset conversation state.

Monitoring, traces & quality scoring

The feedback layer: quality scoring on every conversation and reasoning logged on every decision. Coverage targets whether the trace is a faithful record of what executed — sufficient for a compliance reviewer to reconstruct why an action was taken — rather than a post-hoc narration.

“Quality scoring on every conversation. Reasoning logged on every decision.” getzowie.com

Mapped capabilities

3 capabilities

Trace fidelity to executed actions
Logged reasoning matches the actual sequence of calls, branches, and outcomes, with no unlogged or fabricated steps.
Decision auditability
A reviewer can reconstruct which policy condition authorized an irreversible action from the trace alone.
Quality scoring consistency
Comparable conversations receive comparable scores, and scoring reflects resolution rather than message volume.

Agent configuration & governance

The build surface where CX self-serves and engineering governs: persona and brand voice, intents, knowledge sources, and flow wiring, plus team-based build via Collaboration Mode. Coverage targets whether configuration changes take effect as specified and whether governance boundaries hold between self-serve and engineering-owned layers.

Mapped capabilities

4 capabilities

Persona and tone adherence
Configured brand voice holds across channels and under emotionally charged or adversarial customer messages.
Knowledge grounding and disambiguation
Answers stay grounded in configured knowledge sources, and the agent asks a clarifying question when intent is genuinely ambiguous.
Intent-to-flow binding
Recognized intents map to the intended flow, including near-neighbor intents such as return versus exchange.
Configuration boundary enforcement
Self-serve configuration cannot alter engineering-governed execution logic or integration contracts.

Coverage is mapped from Zowie's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Zowie test?+

The coverage map is generated from Zowie's own public product surface (enterprise customer service AI agent platform): 6 scoring areas — Deterministic execution & Decision Engine, Regulated workflow correctness, and Cross-channel agent continuity, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Zowie evals scored?+

Every case generated for Zowie — across Deterministic execution & Decision Engine and Regulated workflow correctness and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Zowie library include?+

The full Zowie library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, Phrasing invariance to execution path and Language/execution boundary integrity under Deterministic execution & Decision Engine); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Zowie or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Zowie areas and set them up in a Corsac workspace, where you can run every test case against Zowie or your own agent with your own data.