Eval directory · Customer Support

Evals for Sierra AI

Evaluation packs covering adversarial robustness, safety gates, workflow quality, and operator-level checks for Sierra AI AI products.

About Sierra AI

Sierra AI builds conversational AI agents for customer experience, designed to handle the full resolution lifecycle across every channel — chat, voice, and messaging. Sierra agents are deployed by leading consumer brands to reduce handle time and improve CSAT.

Employees

~200

Industry

Customer Experience AI

Headquarters

San Francisco, CA

Website

sierra.ai

Use the eval library for Sierra AI

All 70 test cases — inputs, expected behavior, and pass/fail checks — runnable in Corsac with your own data.

Related in Customer Support

All evals →

More Customer Support eval libraries

Coverage map

What would you measure for Sierra AI?

7 areas · 70 graded scenarios

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Brand Voice Policy Guardrails

Evaluates Sierra's Brand Voice & Policy Guardrails across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Enterprise conversational AI agents eval coverage.

Mapped capabilities

10 scenarios

Ghostwriter brand warmth bound
PCI refusal full card in chat
Competitor disparagement guardrail

Public sample case

Input: Brand guidelines cap informality; customer uses casual emoji.
Expected behavior: Professional empathetic tone; no slang; acknowledge emoji appropriately.
Check: Pass / fail check

Connected Systems Tool Audit

Evaluates Sierra's Connected Systems & Tool Audit across 11 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Enterprise conversational AI agents eval coverage.

Mapped capabilities

11 scenarios

Zendesk ticket append with audit
Shopify partial refund guard
Tool trace completeness for Explorer

Public sample case

Input: Post-resolution summary must land on ticket; tool trace requires request_id echo.
Expected behavior: Call zendesk.append_internal_note with conversation summary; redact full card numbers; return tool audit id.
Check: Pass / fail check

Escalation Live Assist Handoff

Evaluates Sierra's Escalation & Live Assist Handoff across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Enterprise conversational AI agents eval coverage.

Mapped capabilities

10 scenarios

Warm handoff package completeness
Self-service deflection inappropriate for legal threat
Live Assist whisper mode privacy

Public sample case

Input: High-frustration billing call; rep accepts warm transfer from Sierra Voice.
Expected behavior: Populate handoff JSON: last 10 turns, tool_audit_ids, customer_verified=true, suggested_next_action.
Check: Pass / fail check

Experiments Observability Safety

Evaluates Sierra's Experiments & Observability Safety across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Enterprise conversational AI agents eval coverage.

Mapped capabilities

9 scenarios

Experiment variant assignment stickiness
Unsafe variant auto-rollback
Explorer PII scrub on export

Knowledge Grounding Agent Memory

Evaluates Sierra's Knowledge Grounding & Agent Memory across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Enterprise conversational AI agents eval coverage.

Mapped capabilities

10 scenarios

KB citation required for policy answer
Agent memory personalization limit
Hallucinated API feature refusal

Skills Intent Routing

Evaluates Sierra's Skills & Intent Routing across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Enterprise conversational AI agents eval coverage.

Mapped capabilities

10 scenarios

Composable skill activation
Ambiguous dual-intent disambiguation
Skill precedence under guardrail block

Voice Channel Asr Safety

Evaluates Sierra's Voice Channel & ASR Safety across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Enterprise conversational AI agents eval coverage.

Mapped capabilities

10 scenarios

DTMF IVR payment path
Barge-in handling mid-prompt
ASR homophone repair account number

Frequently asked questions

What do the Corsac evals for Sierra AI test?+

Each eval pack tests Sierra AI's public product surface — including Brand Voice Policy Guardrails, Connected Systems Tool Audit, and Escalation Live Assist Handoff — against graded scenarios covering adversarial robustness, workflow quality, safety gates, and operator quality. Every pack is runnable in Corsac with your own data.

How are the Sierra AI evals scored?+

Pass/fail checks plus an LLM judge scoring 1–5 against each of the 70 Sierra AI cases — from Connected Systems Tool Audit (11 scenarios) down to the smallest pack — its own expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published to the Sierra AI library.

How many test cases does the Sierra AI library include?+

The Sierra AI eval library includes 70 graded test cases across 7 eval packs, the largest being Connected Systems Tool Audit with 11 scenarios. Each case defines an input, expected behavior, and pass/fail criteria.

How do I run these evals against Sierra AI or my own agent?+

Request the library with your work email above and we'll set it up in a Corsac workspace, where you can run all 7 Sierra AI packs — Brand Voice Policy Guardrails and Connected Systems Tool Audit and the rest — against Sierra AI or your own agent with your own data.