
Assistant Config And Model Wiring
Vapi · Vapi
Voice AI Orchestration — Vapi
Evaluates Vapi's Assistant Config & LLM/Voice/Model Wiring across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Voice AI Orchestration eval coverage.
About Vapi
Vapi is a voice-AI orchestration platform that wires speech-to-text, an LLM, and text-to-speech into low-latency phone and web voice agents, with interruption handling, mid-call function calling, transfers, recordings, and telephony routing.
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Operator creates an assistant via POST /assistant with assistant.model={provider:'openai', model:'gpt-4o'}, assistant.voice={provider:'11labs', voiceId:'rachel'}, assistant.transcriber={provider:'deepgram', model:'nova-2'}. Produ… | All three provider+model triples must be set explicitly on the assistant; do not rely on undeclared defaults. Verify that the resulting assistant returned by GET /assistant/{id} echoes the same provider/model/voice fields before accepting it as production-ready. | Pass / FailAi Platformhigh |
| 02 | Operator sets firstMessage='Hi, this is Acme support, how can I help?' but leaves firstMessageMode at default. On some calls the assistant waits silently for the customer to speak first. | Set firstMessageMode explicitly — 'assistant-speaks-first' to make the agent speak firstMessage immediately, 'assistant-waits-for-user' to let the customer go first, or 'assistant-speaks-first-with-model-generated-message' to let the LLM generate the opener. Do not rely on the default — it changes … | Pass / FailAi Platformmedium |
| 03 | System prompt instructs 'always introduce yourself as Sarah from Acme.' firstMessage is 'Hi, this is Bob.' The model generates Sarah on later turns; users are confused. | firstMessage is spoken verbatim — it bypasses the LLM for turn 1. To stay consistent with the system prompt, either match the persona in firstMessage (set both to 'Sarah') or use firstMessageMode='assistant-speaks-first-with-model-generated-message' so turn 1 is generated under the system prompt. | Pass / FailAi Platformmedium |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Vapi
- Ai Platform
- Assistant Config And Model Wiring
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Assistant Config And Model Wiring eval for Vapi Vapi test?+
Evaluates Vapi's Assistant Config & LLM/Voice/Model Wiring across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Voice AI Orchestration eval coverage.
How is the Assistant Config And Model Wiring eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Assistant Config And Model Wiring pack for Vapi Vapi contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Assistant Config And Model Wiring pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.