Auth Rate Limits Concurrency Governance
AssemblyAI (Universal-2 + LeMUR) · AssemblyAI
Speech AI Platform — AssemblyAI
Evaluates AssemblyAI's Auth, Rate Limits, Concurrency & Governance across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Speech AI Platform eval coverage.
About AssemblyAI
AssemblyAI is a speech-AI platform with Universal-2 speech-to-text, real-time streaming, Speaker Diarization, Audio Intelligence (summarization, sentiment, content moderation), and LeMUR — an LLM framework that runs over transcripts (task, summary, question-answer, action items).
Sample tests· showing 3 of 10
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Agent passes Authorization: Bearer <api_key> on POST /v2/transcript thinking AssemblyAI uses standard Bearer auth. Server returns 401. | AssemblyAI's Authorization header carries the raw API key (no 'Bearer ' prefix). Set Authorization: <api_key>. Do not assume Bearer is universal; different vendors differ. Read the auth doc and pin the header shape in code. | Pass / FailAi Platformcritical |
| 02 | Burst of 200 simultaneous transcript submissions returns 429s with Retry-After headers on a subset. | Honor Retry-After (seconds or HTTP date). Cap concurrency client-side to stay under documented tier limits. Use exponential backoff with jitter for subsequent retries. Do not retry without backoff — that escalates the throttle window. | Pass / FailAi Platformhigh |
| 03 | Voice-agent platform tries to open 500 concurrent /v2/realtime/ws sessions and gets WS handshake rejections beyond a tier-dependent cap. | Tier-dependent concurrent session cap applies [REQUIRES-VERIFICATION on numeric caps]. Track in-flight session count client-side; back off or queue new sessions when at cap. Surface clearly to operator when capacity is exhausted. Coordinate with AssemblyAI sales for tier increases. | Pass / FailAi Platformhigh |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Assemblyai
- Ai Platform
- Auth Rate Limits Concurrency Governance
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Auth Rate Limits Concurrency Governance eval for AssemblyAI AssemblyAI (Universal-2 + LeMUR) test?+
Evaluates AssemblyAI's Auth, Rate Limits, Concurrency & Governance across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Speech AI Platform eval coverage.
How is the Auth Rate Limits Concurrency Governance eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Auth Rate Limits Concurrency Governance pack for AssemblyAI AssemblyAI (Universal-2 + LeMUR) contains 10 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Auth Rate Limits Concurrency Governance pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.