Eval Library
V
For VapiAI Platform

Assistant Config And Model Wiring

Vapi · Vapi

Voice AI Orchestration — Vapi

Vapi evals — Assistant Config & LLM/Voice/Model Wiring (relift v3 InfraRed)

About Vapi

Vapi is a voice-AI orchestration platform that wires speech-to-text, an LLM, and text-to-speech into low-latency phone and web voice agents, with interruption handling, mid-call function calling, transfers, recordings, and telephony routing.

Employees

~50

Industry

Voice AI Orchestration

Headquarters

San Francisco, CA

Website

vapi.ai

Sample tests· showing 3 of 9

#InputExpected behaviorCheck
01

Operator creates an assistant via POST /assistant with assistant.model={provider:'openai', model:'gpt-4o'}, assistant.voice={provider:'11labs', voiceId:'rachel'}, assistant.transcriber={provider:'deepgram', model:'nova-2'}. Produ…

All three provider+model triples must be set explicitly on the assistant; do not rely on undeclared defaults. Verify that the resulting assistant returned by GET /assistant/{id} echoes the same provider/model/voice fields before accepting it as production-ready.

Pass / FailAi Platformhigh
02

Operator sets firstMessage='Hi, this is Acme support, how can I help?' but leaves firstMessageMode at default. On some calls the assistant waits silently for the customer to speak first.

Set firstMessageMode explicitly — 'assistant-speaks-first' to make the agent speak firstMessage immediately, 'assistant-waits-for-user' to let the customer go first, or 'assistant-speaks-first-with-model-generated-message' to let the LLM generate the opener. Do not rely on the default — it changes …

Pass / FailAi Platformmedium
03

Account-level serverUrl points to https://acme.example/vapi. Assistant overrides with assistant.serverUrl=https://acme.example/vapi-staging. Production traffic hits staging.

Assistant-level serverUrl overrides account-level for that assistant's events (assistant-request, tool-calls, status-update, end-of-call-report). Audit which level is set per environment; never leave a staging override on a production assistant. Round-trip GET /assistant/{id} to confirm the effecti…

Pass / FailAi Platformcritical

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Vapi
  • Ai Platform
  • Assistant Config And Model Wiring

Recommended for

VapiVapi customers

Works with

Related evals

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.