
Lemur
AssemblyAI (Universal-2 + LeMUR) · AssemblyAI
Speech AI Platform — AssemblyAI
Evaluates AssemblyAI's LeMUR across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Speech AI Platform eval coverage.
About AssemblyAI
AssemblyAI is a speech-AI platform with Universal-2 speech-to-text, real-time streaming, Speaker Diarization, Audio Intelligence (summarization, sentiment, content moderation), and LeMUR — an LLM framework that runs over transcripts (task, summary, question-answer, action items).
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Agent calls POST /lemur/v3/generate/task with both transcript_ids=['t_1','t_2'] and input_text='...' for the same prompt. | Use transcript_ids[] when the LLM should run over AssemblyAI-produced transcripts; use input_text for arbitrary text. Combining both has documented precedence — verify the docs and pick one cleanly. Multi-transcript LeMUR concatenates transcripts under a context budget set by final_model's context … | Pass / FailAi Platformhigh |
| 02 | Agent wants Claude 3.5 Sonnet specifically for a LeMUR summary task. Sets final_model='anthropic/claude-3-5-sonnet'. | final_model selects the LLM backing LeMUR (default | basic | anthropic/* per current docs). Pin a model id you've evaluated; do not rely on 'default' across releases as the underlying model may shift. Verify available model ids in current docs [REQUIRES-VERIFICATION]. | Pass / FailAi Platformmedium |
| 03 | Agent calls /lemur/v3/generate/question-answer with questions=[{question:'What was the meeting outcome?', answer_format:'short sentence', answer_options:['signed','postponed','rejected']}]. | answer_format constrains the answer shape (free text); answer_options forces a choice from a closed set. For categorical extraction always pass answer_options. Validate the returned answer matches an option exactly (case-insensitive); reject responses that drift outside the option list. | Pass / FailAi Platformhigh |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Assemblyai
- Ai Platform
- Lemur
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Lemur eval for AssemblyAI AssemblyAI (Universal-2 + LeMUR) test?+
Evaluates AssemblyAI's LeMUR across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Speech AI Platform eval coverage.
How is the Lemur eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Lemur pack for AssemblyAI AssemblyAI (Universal-2 + LeMUR) contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Lemur pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.