Realtime Voice And Turn Taking
Vapi · Vapi
Voice AI Orchestration — Vapi
Evaluates Vapi's Real-time Voice & Turn-taking across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Voice AI Orchestration eval coverage.
About Vapi
Vapi is a voice-AI orchestration platform that wires speech-to-text, an LLM, and text-to-speech into low-latency phone and web voice agents, with interruption handling, mid-call function calling, transfers, recordings, and telephony routing.
Sample tests· showing 3 of 10
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Default startSpeakingPlan.waitSeconds is short. On a noisy call, the assistant starts speaking over the customer's last syllable on most turns. | Raise startSpeakingPlan.waitSeconds (e.g., 0.4 → 0.8) and enable transcriptionEndpointingPlan.smartEndpointingEnabled if available so end-of-turn is detected from transcript shape, not just silence. Validate by measuring interruption rate in artifact.messages[] before/after. | Pass / FailAi Platformhigh |
| 02 | Long customer monologue — assistant stays silent for 30 seconds. Customer says 'are you still there?' | Enable backchanneling (mm-hmm / right / I see) on the assistant so the agent vocalizes presence without taking the turn. Configure acknowledgementPhrases or rely on the documented default — verify in artifact.messages[] that backchannel tokens fire mid-monologue, not after. | Pass / FailAi Platformmedium |
| 03 | Customer says 'wait —' and the assistant keeps talking for another 4 seconds before stopping. Operator wants snappier interruption. | Lower stopSpeakingPlan.numWords (e.g., 2 → 1) so the assistant aborts TTS after the first user word, and verify stopSpeakingPlan.voiceSeconds and backoffSeconds are low. Test with deliberate barge-ins; measure assistant.speech.interrupted hook firing latency. | Pass / FailAi Platformhigh |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Vapi
- Ai Platform
- Realtime Voice And Turn Taking
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Realtime Voice And Turn Taking eval for Vapi Vapi test?+
Evaluates Vapi's Real-time Voice & Turn-taking across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Voice AI Orchestration eval coverage.
How is the Realtime Voice And Turn Taking eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Realtime Voice And Turn Taking pack for Vapi Vapi contains 10 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Realtime Voice And Turn Taking pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.