Eval Library
A
For AssemblyAIAI Platform

Webhooks And Async Delivery

AssemblyAI (Universal-2 + LeMUR) · AssemblyAI

Speech AI Platform — AssemblyAI

AssemblyAI evals — Webhooks & Async Delivery (relift v3 InfraRed)

About AssemblyAI

AssemblyAI is a speech-AI platform with Universal-2 speech-to-text, real-time streaming, Speaker Diarization, Audio Intelligence (summarization, sentiment, content moderation), and LeMUR — an LLM framework that runs over transcripts (task, summary, question-answer, action items).

Employees

~150

Industry

Speech AI

Headquarters

San Francisco, CA

Sample tests· showing 3 of 9

#InputExpected behaviorCheck
01

Agent submits transcripts with webhook_url=https://app.example.com/aai-hook, webhook_auth_header_name='X-AAI-Secret', webhook_auth_header_value='<long random secret>'.

AssemblyAI POSTs to webhook_url with the configured header name carrying the configured value verbatim. Receiver must check the header name AND value match the expected shared secret before processing. Treat any POST without the header as untrusted.

Pass / FailAi Platformcritical
02

Receiver returns 502 on first webhook delivery due to a deploy. AssemblyAI retries per its documented retry policy.

AssemblyAI retries non-2xx deliveries over a documented window [REQUIRES-VERIFICATION on exact retry count and interval]. Receiver should return 2xx promptly for accepted events even if downstream processing is async — do not hold the connection for downstream completion.

Pass / FailAi Platformhigh
03

Webhook body includes transcript_id and status (completed | error). Receiver assumes the body also has full transcript text.

Webhook payload carries identity and status; fetch the full transcript via GET /v2/transcript/{id} after receipt. Treat the webhook as a notification, not as the transcript itself. This decouples webhook size from transcript size and improves reliability.

Pass / FailAi Platformmedium

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Assemblyai
  • Ai Platform
  • Webhooks And Async Delivery

Recommended for

AssemblyAI (Universal-2 + LeMUR)AssemblyAI customers

Works with

Related evals

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.