Customer Lab Data Delivery
Mercor · Mercor
AI Talent Marketplace & Data Labeling — Mercor
Evaluates Mercor's Customer / Lab Data Delivery across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Talent Marketplace & Data Labeling eval coverage.
About Mercor
Mercor is an AI talent marketplace and human-data infrastructure provider for frontier AI labs and enterprises. It runs ~20-minute AI-led video interviews, matches a global network of domain experts to projects, and operates labeling, RLHF preference data, rubric authoring, and evaluation framework workflows for customers including leading AI labs.
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Customer lab's DPA specifies JSONL with documented schema. Mercor ships a Parquet file because 'JSONL was too large.' | Delivery format and schema are contractual. Format changes require explicit customer sign-off; do not unilaterally substitute a format because of operational convenience. Document the schema version in delivery metadata and validate every shipped file against it before release. | Pass / FailAi Platformhigh |
| 02 | Delivery is hosted at a signed S3 URL valid for 7 days. The lab fetches on day 8 and gets a 403. | Surface signed-URL expiry as a delivery metadata field; on access denial post-expiry, re-issue a fresh signed URL via an authenticated endpoint rather than re-uploading. Document the retention policy on the source object (does Mercor delete after N days?). | Pass / FailAi Platformmedium |
| 03 | A delivery includes 100k rows. Lab's downstream pipeline reports 'row 47281 appears corrupted.' Mercor cannot tell whether corruption is in transit or at-rest. | Compute and ship per-row integrity hashes (or a per-file hash for small batches); lab can verify on receipt. Provide a manifest with row counts and hashes. On corruption report, replay the source partition to isolate. | Pass / FailAi Platformhigh |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Mercor
- Ai Platform
- Customer Lab Data Delivery
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Customer Lab Data Delivery eval for Mercor Mercor test?+
Evaluates Mercor's Customer / Lab Data Delivery across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Talent Marketplace & Data Labeling eval coverage.
How is the Customer Lab Data Delivery eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Customer Lab Data Delivery pack for Mercor Mercor contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Customer Lab Data Delivery pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.