Streaming Stt Websocket
Deepgram Speech AI Platform · Deepgram
Speech AI Platform — Deepgram (Nova STT, Aura TTS, Voice Agent)
Evaluates Deepgram's Streaming STT (WebSocket) across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Speech AI Platform eval coverage.
About Deepgram
Deepgram is a speech-AI platform offering streaming and batch speech-to-text (Nova), Aura text-to-speech, speaker diarization, redaction, and smart formatting across 30+ languages — used by voice-agent platforms, contact centers, and media teams.
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Agent opens wss://api.deepgram.com/v1/listen?model=nova-3&encoding=linear16&sample_rate=16000 with header 'Authorization: Token <DEEPGRAM_API_KEY>'. | Send the Authorization header in the WebSocket upgrade request — Deepgram uses 'Token <key>' scheme, not 'Bearer'. Confirm 101 Switching Protocols before streaming audio frames. Treat 401/403 as a credential issue and surface to operator; do not retry the same key on auth failure. | Pass / FailAi Platformcritical |
| 02 | Connection uses interim_results=true. Server emits a sequence of is_final=false hypotheses followed by an is_final=true segment. | Render interim (is_final=false) results to a transient overlay; commit only is_final=true messages to the persisted transcript. Do not double-commit when the final arrives — replace the matching interim span, not append. | Pass / FailAi Platformhigh |
| 03 | Agent sets endpointing=300 and utterance_end_ms=1000 expecting a UtteranceEnd event when the speaker pauses ~1s. | utterance_end_ms requires vad_events=true (or interim_results=true per docs). When the silence threshold elapses, server emits type='UtteranceEnd' with a last_word_end timestamp — use it to flush the partial buffer downstream, do not wait for the next final. | Pass / FailAi Platformhigh |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Deepgram
- Ai Platform
- Streaming Stt Websocket
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Streaming Stt Websocket eval for Deepgram Deepgram Speech AI Platform test?+
Evaluates Deepgram's Streaming STT (WebSocket) across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Speech AI Platform eval coverage.
How is the Streaming Stt Websocket eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Streaming Stt Websocket pack for Deepgram Deepgram Speech AI Platform contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Streaming Stt Websocket pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.