Speech Whisper Stt
GroqCloud API · Groq
Fast Inference — Groq (GroqCloud)
Evaluates Groq's Speech (Whisper STT) across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Fast Inference eval coverage.
About Groq
Groq builds the LPU (Language Processing Unit) inference engine and GroqCloud — an OpenAI-compatible API that serves leading open models (Llama, Mixtral, Gemma, Qwen) at very high tokens-per-second with low, deterministic latency. Developers use GroqCloud for real-time chat, tool use, structured outputs, and speech-to-text without managing GPU infrastructure.
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Agent posts an audio file to /v1/chat/completions expecting transcription, instead of the audio transcription endpoint. | Send audio to POST /openai/v1/audio/transcriptions with a Whisper model id (e.g. whisper-large-v3) and the file as multipart form data. Transcription is a distinct endpoint from chat completions; confirm the exact model id and endpoint against current docs. | Pass / FailAi Platformhigh |
| 02 | A workflow needs Spanish audio rendered as English text, but the agent calls /audio/transcriptions and gets Spanish text. | Use /audio/transcriptions to transcribe speech in its original language; use /audio/translations to translate non-English speech into English text. Pick the endpoint that matches the desired output language. | Pass / FailAi Platformmedium |
| 03 | Audio is known to be German, but the agent omits the language parameter and Whisper occasionally misdetects the language, hurting accuracy. | Pass the language parameter (ISO code) when the source language is known to improve accuracy and latency. Only rely on auto-detection when the language is genuinely unknown. A wrong hint, however, can hurt accuracy — set it only when confident. | Pass / FailAi Platformmedium |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Groq
- Ai Platform
- Speech Whisper Stt
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Speech Whisper Stt eval for Groq GroqCloud API test?+
Evaluates Groq's Speech (Whisper STT) across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Fast Inference eval coverage.
How is the Speech Whisper Stt eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Speech Whisper Stt pack for Groq GroqCloud API contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Speech Whisper Stt pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.