Webhooks And Async Delivery
AssemblyAI (Universal-2 + LeMUR) · AssemblyAI
Speech AI Platform — AssemblyAI
Evaluates AssemblyAI's Webhooks & Async Delivery across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Speech AI Platform eval coverage.
About AssemblyAI
AssemblyAI is a speech-AI platform with Universal-2 speech-to-text, real-time streaming, Speaker Diarization, Audio Intelligence (summarization, sentiment, content moderation), and LeMUR — an LLM framework that runs over transcripts (task, summary, question-answer, action items).
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Agent submits transcripts with webhook_url=https://app.example.com/aai-hook, webhook_auth_header_name='X-AAI-Secret', webhook_auth_header_value='<long random secret>'. | AssemblyAI POSTs to webhook_url with the configured header name carrying the configured value verbatim. Receiver must check the header name AND value match the expected shared secret before processing. Treat any POST without the header as untrusted. | Pass / FailAi Platformcritical |
| 02 | Receiver returns 502 on first webhook delivery due to a deploy. AssemblyAI retries per its documented retry policy. | AssemblyAI retries non-2xx deliveries over a documented window [REQUIRES-VERIFICATION on exact retry count and interval]. Receiver should return 2xx promptly for accepted events even if downstream processing is async — do not hold the connection for downstream completion. | Pass / FailAi Platformhigh |
| 03 | Webhook for transcript_id=t_abc arrives twice (retry on a 5xx that actually succeeded). Receiver writes to DB on each. | Use transcript_id as the idempotency key in the receiver. Upsert by transcript_id; do not blindly insert. Drop or no-op on duplicate. Webhooks are at-least-once — design accordingly. | Pass / FailAi Platformhigh |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Assemblyai
- Ai Platform
- Webhooks And Async Delivery
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Webhooks And Async Delivery eval for AssemblyAI AssemblyAI (Universal-2 + LeMUR) test?+
Evaluates AssemblyAI's Webhooks & Async Delivery across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Speech AI Platform eval coverage.
How is the Webhooks And Async Delivery eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Webhooks And Async Delivery pack for AssemblyAI AssemblyAI (Universal-2 + LeMUR) contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Webhooks And Async Delivery pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.