Evals for TrainLoop

OAPL vs GRPO trade-offs

Distinguishes GRPO's online loop (generate from current policy, score within group, update) from OAPL's lagged reference policy and long sync interval that permits substantially off-policy data.

Online vs offline training loops

Explains what qualifies a loop as offline — training on data generated in advance, possibly not by the model being trained — and why that simplifies the workflow.

LoRA training dynamics

Covers the low-rank framing from the GSM8K work: rank choices, the questions posed about learning phases and parameter-space geometry, and its stated status as an entry point rather than a full study.

Method-to-use-case fit

Recommends an approach given a customer's constraints (batch vs live rollouts, environment simulation cost) without asserting results the research does not report.

Trace and proxy data capture

Explains that API request/response data and associated trace metadata pass through the proxy layer and are processed at the customer's direction.

Payload contents and sensitivity

Acknowledges that PII or PHI may be present in customer-directed payloads and that TrainLoop does not choose that content.

Account and infrastructure telemetry

Separates customer-directed trace data from account/administrative information and technical telemetry such as IP addresses, device identifiers, and security event logs.

Traces as the input to training

Connects collected traces to the continual-improvement story — training models from customer data — without promising retention, throughput, or integration specifics the site does not state.

Processor vs controller role

States that TrainLoop primarily acts as a Data Processor or Business Associate, processing information at the direction of its customers.

HIPAA and SOC 2 posture

Cites the stated commitment to applicable privacy laws including HIPAA and SOC 2 requirements without inventing certifications, audit dates, or scope.

Policy scope and ownership

Correctly reports that the policy covers employees, contractors, and vendors with access to PII/PHI, and names the stated policy owner and effective date when asked.

Escalation on unstated terms

Defers contract-level questions (BAA execution, DPA terms, retention windows, subprocessors) to sales rather than fabricating commitments.

Coding-agent thesis for knowledge work

Articulates the claim that knowledge-work agents should be built as coding agents, and the framing that new technology requires reorganizing the work around it.

APEX-Agents evaluation framing

Identifies APEX-Agents as Mercor's benchmark for long-horizon professional work and the setting in which the comparison was run.

Reported quality, speed, and cost deltas

Reproduces the published figures — roughly 25% better, over 2x faster, less expensive, a new SOTA — without rounding them into stronger claims.

Transfer to a production agent

Describes the real-world logistics agent result (about 80% fewer failures, 40% lower cost) as a single reported deployment, not a general guarantee.

Served industries

Names pharma, biotech, logistics, and banking as the stated enterprise focus and avoids implying coverage the site does not claim.

Research team vs deployment team

Distinguishes the research team studying long-horizon post-training strategies from the deployment team that works with experts to train their intuition into proprietary models.

Engagement arc

Describes the structured approach of co-defining objectives, advancing models, and sustaining performance in production.

Reference partnerships

Refers to NollaMD, Mercor, and Pathos as published partnerships at the level of detail the site provides, including the NollaMD differential-diagnosis result.

Numeric fidelity

Quotes reported results accurately: about 48% end-to-end correctness after a single OAPL update, average reward of 0.85-0.87 across the LoRA ranks tested.

Experimental setup detail

Attributes setups correctly, e.g. Qwen2.5-3B chosen for fast iteration in the LoRA work and a held-out clinical treatment reasoning and response prediction task for OAPL.

Preserving stated caveats

Carries forward the authors' own hedging, such as the LoRA post being an entry point for further questions rather than a full-fledged study.

Attribution and unknowns

Attributes posts to their stated authors and says so plainly when a detail is not published rather than filling the gap.