Webhooks And Async Delivery
AssemblyAI (Universal-2 + LeMUR) · AssemblyAI
Speech AI Platform — AssemblyAI
AssemblyAI evals — Webhooks & Async Delivery (relift v3 InfraRed)
About AssemblyAI
AssemblyAI is a speech-AI platform with Universal-2 speech-to-text, real-time streaming, Speaker Diarization, Audio Intelligence (summarization, sentiment, content moderation), and LeMUR — an LLM framework that runs over transcripts (task, summary, question-answer, action items).
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Agent submits transcripts with webhook_url=https://app.example.com/aai-hook, webhook_auth_header_name='X-AAI-Secret', webhook_auth_header_value='<long random secret>'. | AssemblyAI POSTs to webhook_url with the configured header name carrying the configured value verbatim. Receiver must check the header name AND value match the expected shared secret before processing. Treat any POST without the header as untrusted. | Pass / FailAi Platformcritical |
| 02 | Receiver returns 502 on first webhook delivery due to a deploy. AssemblyAI retries per its documented retry policy. | AssemblyAI retries non-2xx deliveries over a documented window [REQUIRES-VERIFICATION on exact retry count and interval]. Receiver should return 2xx promptly for accepted events even if downstream processing is async — do not hold the connection for downstream completion. | Pass / FailAi Platformhigh |
| 03 | Webhook body includes transcript_id and status (completed | error). Receiver assumes the body also has full transcript text. | Webhook payload carries identity and status; fetch the full transcript via GET /v2/transcript/{id} after receipt. Treat the webhook as a notification, not as the transcript itself. This decouples webhook size from transcript size and improves reliability. | Pass / FailAi Platformmedium |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Assemblyai
- Ai Platform
- Webhooks And Async Delivery
Recommended for
Works with
Related evals
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.