Assistant Config And Model Wiring
Vapi · Vapi
Voice AI Orchestration — Vapi
Vapi evals — Assistant Config & LLM/Voice/Model Wiring (relift v3 InfraRed)
About Vapi
Vapi is a voice-AI orchestration platform that wires speech-to-text, an LLM, and text-to-speech into low-latency phone and web voice agents, with interruption handling, mid-call function calling, transfers, recordings, and telephony routing.
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Operator creates an assistant via POST /assistant with assistant.model={provider:'openai', model:'gpt-4o'}, assistant.voice={provider:'11labs', voiceId:'rachel'}, assistant.transcriber={provider:'deepgram', model:'nova-2'}. Produ… | All three provider+model triples must be set explicitly on the assistant; do not rely on undeclared defaults. Verify that the resulting assistant returned by GET /assistant/{id} echoes the same provider/model/voice fields before accepting it as production-ready. | Pass / FailAi Platformhigh |
| 02 | Operator sets firstMessage='Hi, this is Acme support, how can I help?' but leaves firstMessageMode at default. On some calls the assistant waits silently for the customer to speak first. | Set firstMessageMode explicitly — 'assistant-speaks-first' to make the agent speak firstMessage immediately, 'assistant-waits-for-user' to let the customer go first, or 'assistant-speaks-first-with-model-generated-message' to let the LLM generate the opener. Do not rely on the default — it changes … | Pass / FailAi Platformmedium |
| 03 | Account-level serverUrl points to https://acme.example/vapi. Assistant overrides with assistant.serverUrl=https://acme.example/vapi-staging. Production traffic hits staging. | Assistant-level serverUrl overrides account-level for that assistant's events (assistant-request, tool-calls, status-update, end-of-call-report). Audit which level is set per environment; never leave a staging override on a production assistant. Round-trip GET /assistant/{id} to confirm the effecti… | Pass / FailAi Platformcritical |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Vapi
- Ai Platform
- Assistant Config And Model Wiring
Recommended for
Works with
Related evals
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.