AssemblyAI
For AssemblyAIAI Platform

Lemur

AssemblyAI (Universal-2 + LeMUR) · AssemblyAI

Speech AI Platform — AssemblyAI

Evaluates AssemblyAI's LeMUR across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Speech AI Platform eval coverage.

About AssemblyAI

AssemblyAI is a speech-AI platform with Universal-2 speech-to-text, real-time streaming, Speaker Diarization, Audio Intelligence (summarization, sentiment, content moderation), and LeMUR — an LLM framework that runs over transcripts (task, summary, question-answer, action items).

Employees

~150

Industry

Speech AI

Headquarters

San Francisco, CA

Sample tests· showing 3 of 9

#InputExpected behaviorCheck
01

Agent calls POST /lemur/v3/generate/task with both transcript_ids=['t_1','t_2'] and input_text='...' for the same prompt.

Use transcript_ids[] when the LLM should run over AssemblyAI-produced transcripts; use input_text for arbitrary text. Combining both has documented precedence — verify the docs and pick one cleanly. Multi-transcript LeMUR concatenates transcripts under a context budget set by final_model's context …

Pass / FailAi Platformhigh
02

Agent wants Claude 3.5 Sonnet specifically for a LeMUR summary task. Sets final_model='anthropic/claude-3-5-sonnet'.

final_model selects the LLM backing LeMUR (default | basic | anthropic/* per current docs). Pin a model id you've evaluated; do not rely on 'default' across releases as the underlying model may shift. Verify available model ids in current docs [REQUIRES-VERIFICATION].

Pass / FailAi Platformmedium
03

Agent calls /lemur/v3/generate/question-answer with questions=[{question:'What was the meeting outcome?', answer_format:'short sentence', answer_options:['signed','postponed','rejected']}].

answer_format constrains the answer shape (free text); answer_options forces a choice from a closed set. For categorical extraction always pass answer_options. Validate the returned answer matches an option exactly (case-insensitive); reject responses that drift outside the option list.

Pass / FailAi Platformhigh

Unlock full benchmark

6 more test cases

Use this benchmark

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Assemblyai
  • Ai Platform
  • Lemur

Recommended for

AssemblyAI (Universal-2 + LeMUR)AssemblyAI customers

Works with

Related evals

Frequently asked questions

What does the Lemur eval for AssemblyAI AssemblyAI (Universal-2 + LeMUR) test?+

Evaluates AssemblyAI's LeMUR across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Speech AI Platform eval coverage.

How is the Lemur eval scored?+

The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

How many test cases does this eval pack include?+

The Lemur pack for AssemblyAI AssemblyAI (Universal-2 + LeMUR) contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.

How do I run this eval?+

Sign up for Corsac, connect your model or agent endpoint, and run the Lemur pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.