Vapi
For VapiAI Platform

Assistant Config And Model Wiring

Vapi · Vapi

Voice AI Orchestration — Vapi

Evaluates Vapi's Assistant Config & LLM/Voice/Model Wiring across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Voice AI Orchestration eval coverage.

About Vapi

Vapi is a voice-AI orchestration platform that wires speech-to-text, an LLM, and text-to-speech into low-latency phone and web voice agents, with interruption handling, mid-call function calling, transfers, recordings, and telephony routing.

Employees

~50

Industry

Voice AI Orchestration

Headquarters

San Francisco, CA

Website

vapi.ai

Sample tests· showing 3 of 9

#InputExpected behaviorCheck
01

Operator creates an assistant via POST /assistant with assistant.model={provider:'openai', model:'gpt-4o'}, assistant.voice={provider:'11labs', voiceId:'rachel'}, assistant.transcriber={provider:'deepgram', model:'nova-2'}. Produ…

All three provider+model triples must be set explicitly on the assistant; do not rely on undeclared defaults. Verify that the resulting assistant returned by GET /assistant/{id} echoes the same provider/model/voice fields before accepting it as production-ready.

Pass / FailAi Platformhigh
02

Operator sets firstMessage='Hi, this is Acme support, how can I help?' but leaves firstMessageMode at default. On some calls the assistant waits silently for the customer to speak first.

Set firstMessageMode explicitly — 'assistant-speaks-first' to make the agent speak firstMessage immediately, 'assistant-waits-for-user' to let the customer go first, or 'assistant-speaks-first-with-model-generated-message' to let the LLM generate the opener. Do not rely on the default — it changes …

Pass / FailAi Platformmedium
03

System prompt instructs 'always introduce yourself as Sarah from Acme.' firstMessage is 'Hi, this is Bob.' The model generates Sarah on later turns; users are confused.

firstMessage is spoken verbatim — it bypasses the LLM for turn 1. To stay consistent with the system prompt, either match the persona in firstMessage (set both to 'Sarah') or use firstMessageMode='assistant-speaks-first-with-model-generated-message' so turn 1 is generated under the system prompt.

Pass / FailAi Platformmedium

Unlock full benchmark

6 more test cases

Use this benchmark

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Vapi
  • Ai Platform
  • Assistant Config And Model Wiring

Recommended for

VapiVapi customers

Works with

Related evals

Frequently asked questions

What does the Assistant Config And Model Wiring eval for Vapi Vapi test?+

Evaluates Vapi's Assistant Config & LLM/Voice/Model Wiring across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Voice AI Orchestration eval coverage.

How is the Assistant Config And Model Wiring eval scored?+

The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

How many test cases does this eval pack include?+

The Assistant Config And Model Wiring pack for Vapi Vapi contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.

How do I run this eval?+

Sign up for Corsac, connect your model or agent endpoint, and run the Assistant Config And Model Wiring pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.