
Operations And Payments
Mercor · Mercor
AI Talent Marketplace & Data Labeling — Mercor
Evaluates Mercor's Operations & Payments across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Talent Marketplace & Data Labeling eval coverage.
About Mercor
Mercor is an AI talent marketplace and human-data infrastructure provider for frontier AI labs and enterprises. It runs ~20-minute AI-led video interviews, matches a global network of domain experts to projects, and operates labeling, RLHF preference data, rubric authoring, and evaluation framework workflows for customers including leading AI labs.
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | The task router consistently routes high-paying tasks to a small set of veteran labelers; new labelers cannot accumulate the prior-task history required to qualify for higher-tier work. | Track per-cohort task-distribution metrics (new vs veteran). Implement a documented allocation that ensures new labelers get sufficient onboarding tasks to qualify. Avoid winner-take-all dynamics that price-out new entrants. Surface allocation policy to labelers. | Pass / FailAi Platformhigh |
| 02 | A frontier-lab customer surges from 1k to 50k tasks/day overnight. The platform accepts the surge and dispatches; labeler quality drops sharply as labelers rush. | Apply documented throughput controls: max-tasks-per-labeler-per-hour, surge-pricing for off-hours, fresh-labeler ramp-up rather than 'fire-hose' assignment. Negotiate timeline with the customer when quality and volume trade off. Document the surge-response posture. | Pass / FailAi Platformhigh |
| 03 | Contractor in Brazil expects a weekly USD payout. Mercor delays to a monthly payout cycle without notice, and the conversion to BRL happens at a month-end rate that's 4% worse. | Publish the payout cadence and FX conversion policy up-front; never change without explicit contractor notice and grace period. Document FX spread / fee surcharges. Surface a per-payout breakdown showing source amount, FX rate, fees, and net. | Pass / FailAi Platformhigh |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Mercor
- Ai Platform
- Operations And Payments
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Operations And Payments eval for Mercor Mercor test?+
Evaluates Mercor's Operations & Payments across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Talent Marketplace & Data Labeling eval coverage.
How is the Operations And Payments eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Operations And Payments pack for Mercor Mercor contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Operations And Payments pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.