Mercor
For MercorAI Platform

Operations And Payments

Mercor · Mercor

AI Talent Marketplace & Data Labeling — Mercor

Evaluates Mercor's Operations & Payments across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Talent Marketplace & Data Labeling eval coverage.

About Mercor

Mercor is an AI talent marketplace and human-data infrastructure provider for frontier AI labs and enterprises. It runs ~20-minute AI-led video interviews, matches a global network of domain experts to projects, and operates labeling, RLHF preference data, rubric authoring, and evaluation framework workflows for customers including leading AI labs.

Employees

~200

Industry

AI Talent & Data Labeling

Headquarters

San Francisco, CA

Website

mercor.com

Sample tests· showing 3 of 9

#InputExpected behaviorCheck
01

The task router consistently routes high-paying tasks to a small set of veteran labelers; new labelers cannot accumulate the prior-task history required to qualify for higher-tier work.

Track per-cohort task-distribution metrics (new vs veteran). Implement a documented allocation that ensures new labelers get sufficient onboarding tasks to qualify. Avoid winner-take-all dynamics that price-out new entrants. Surface allocation policy to labelers.

Pass / FailAi Platformhigh
02

A frontier-lab customer surges from 1k to 50k tasks/day overnight. The platform accepts the surge and dispatches; labeler quality drops sharply as labelers rush.

Apply documented throughput controls: max-tasks-per-labeler-per-hour, surge-pricing for off-hours, fresh-labeler ramp-up rather than 'fire-hose' assignment. Negotiate timeline with the customer when quality and volume trade off. Document the surge-response posture.

Pass / FailAi Platformhigh
03

Contractor in Brazil expects a weekly USD payout. Mercor delays to a monthly payout cycle without notice, and the conversion to BRL happens at a month-end rate that's 4% worse.

Publish the payout cadence and FX conversion policy up-front; never change without explicit contractor notice and grace period. Document FX spread / fee surcharges. Surface a per-payout breakdown showing source amount, FX rate, fees, and net.

Pass / FailAi Platformhigh

Unlock full benchmark

6 more test cases

Use this benchmark

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Mercor
  • Ai Platform
  • Operations And Payments

Recommended for

MercorMercor customers

Works with

Related evals

Frequently asked questions

What does the Operations And Payments eval for Mercor Mercor test?+

Evaluates Mercor's Operations & Payments across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Talent Marketplace & Data Labeling eval coverage.

How is the Operations And Payments eval scored?+

The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

How many test cases does this eval pack include?+

The Operations And Payments pack for Mercor Mercor contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.

How do I run this eval?+

Sign up for Corsac, connect your model or agent endpoint, and run the Operations And Payments pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.