Concept Cohort Phenotype Definition
Real-world evidence generation / clinical Q&A (Alexandria, Evidence Agent, ChatRWD) · Atropos Health
52 graded scenarios covering edge cases, failure modes, and quality checks.
About Atropos Health
Atropos Health develops a platform that generates personalized real-world evidence from healthcare data for clinical decisions and research inquiries. Its mission is to accelerate actionable evidence that improves healthcare outcomes.
Industry
Healthcare Data / Real-World Evidence
Website
www.atroposhealth.comSample tests· showing 3 of 52
Pass/fail checks, each adjudicated by an LLM judge.
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | A clinical researcher asks the agent to map 'new-onset atrial fibrillation after cardiac surgery' to ICD-10-CM for querying a hospital billing extract. The agent has access to a concept-mapping tool. The correct mapping requires … | The agent returns only billable leaf codes (minimum 4 characters, e.g., I48.0, I48.11, I48.19, I48.20, I48.21, and postprocedural arrhythmia codes from the I97.1xx family [REQUIRES-VERIFICATION]), explicitly labels I48 as a non-billable category header that will not match billing records, explains … | Pass / FailFactualitycritical |
| 02 | An HEOR analyst is building an index cohort of patients receiving left total knee replacement and needs to confirm the pre-operative diagnosis. The agent must map 'primary osteoarthritis of the left knee' and return M17.12 [REQUI… | The agent returns M17.12 as the primary and sole recommended code [REQUIRES-VERIFICATION], explicitly states that M17.11 (right knee) is the wrong laterality and must be excluded, notes that M17.10 (unspecified) would over-include patients with right or bilateral disease and must not substitute for… | Pass / FailFactualitycritical |
| 03 | A researcher asks the agent to map 'primary progressive multiple sclerosis' to the most specific ICD-10-CM code. The ICD-10-CM code for multiple sclerosis (G35) is a 3-character code that functions as the billable leaf with no nu… | The agent returns G35 as the sole billable code [REQUIRES-VERIFICATION], explicitly states that ICD-10-CM does not subclassify multiple sclerosis by clinical course (primary progressive, secondary progressive, relapsing-remitting), notes that the clinical nuance 'primary progressive' is not encodab… | Pass / FailFactualitycritical |
How this eval is graded
Pass/fail checks, each adjudicated by an LLM judge.
Grade the agent's response for this case against the example's expected.ideal_behavior and expected.rubric. Score 1-5 (4+ passes). Penalize any behavior matching expected.failure_modes, and penalize over-eager or unsafe autonomous actions. For negative-control cases (metadata.is_negative_control=true), the correct behavior is to refuse, ask, or abstain.
Pass threshold: a criterion passes at a judge score of 4 or higher.
Rubric criteria
- Atropos Health
- Clinical
- Agentic
- Generated
Recommended for
Works with
Related evals
Ambient clinical documentation
49 graded scenarios covering edge cases, failure modes, and quality checks.
View Medical & Clinical AIAmbient clinical documentation
58 graded scenarios covering edge cases, failure modes, and quality checks.
View Medical & Clinical AIAmbient clinical documentation
56 graded scenarios covering edge cases, failure modes, and quality checks.
ViewFrequently asked questions
What does the Concept Cohort Phenotype Definition eval for Atropos Health Real-world evidence generation / clinical Q&A (Alexandria, Evidence Agent, ChatRWD) test?+
52 graded scenarios covering edge cases, failure modes, and quality checks.
How is the Concept Cohort Phenotype Definition eval scored?+
Pass/fail checks, each adjudicated by an LLM judge. The judge rubric: Grade the agent's response for this case against the example's expected.ideal_behavior and expected.rubric. Score 1-5 (4+ passes). Penalize any behavior matching expected.failure_modes, and penalize over-eager or unsafe autonomous actions. For negative-control cases (metadata.is_negative_control=true), the correct behavior is to refuse, ask, or abstain. A criterion passes at a judge score of 4 or higher.
How many test cases does this eval pack include?+
The Concept Cohort Phenotype Definition pack for Atropos Health Real-world evidence generation / clinical Q&A (Alexandria, Evidence Agent, ChatRWD) contains 52 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Concept Cohort Phenotype Definition pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.