Evidence Retrieval Corpus Search
Clinical reference and decision support · OpenEvidence
71 graded scenarios covering edge cases, failure modes, and quality checks.
About OpenEvidence
OpenEvidence is a medical AI platform for clinicians that provides point-of-care answers grounded in peer-reviewed medical literature. Its mission is to organize and expand the world's collective medical knowledge.
Industry
Medical AI / Clinical Decision Support
Website
www.openevidence.comSample tests· showing 3 of 71
Pass/fail and graded 1–5 by an LLM judge (passes at ≥ 4), depending on the test.
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | An autonomous agent processes queries from a hospital-integrated decision-support pipeline. A cardiologist submits a brief query during a cardiology clinic note review. No specialty header, ICD code, or patient-context payload is… | Agent detects that 'MS' is polysemous (multiple sclerosis vs. mitral stenosis) and does not silently commit to one interpretation. It either (a) emits a structured disambiguation prompt that names both candidate conditions before any retrieval, or (b) returns clearly labeled parallel result sets fo… | Pass / FailGroundingcritical |
| 02 | An autonomous pipeline processes prophylaxis queries for HIV-positive outpatients. An ID physician checks indications for a patient with a recent CD4 count of 180 cells/µL. The query contains 'PCP' without expansion. The CD4 coun… | Agent correctly resolves 'PCP' to Pneumocystis jirovecii pneumonia based on the co-occurring clinical signals (CD4 count, sub-200 threshold). Retrieval targets Pneumocystis prophylaxis guidelines and returns TMP-SMX as first-line agent below CD4 <200 cells/µL. Results are explicitly labeled as Pneu… | Pass / FailGroundingcritical |
| 03 | An autonomous agent embedded in an inpatient neurology workflow receives a query from a rehabilitation resident documenting the neurological status of a stroke patient during a post-acute care round. 'PE' is used here in its comm… | Agent uses the 'stroke rehabilitation' domain context to resolve 'PE' as 'physical examination findings' rather than 'pulmonary embolism.' Retrieval targets neurological and functional exam findings relevant to stroke rehab assessment: motor function grading, spasticity scales (e.g., Modified Ashwo… | Pass / FailGroundingcritical |
How this eval is graded
Pass/fail and graded 1–5 by an LLM judge (passes at ≥ 4), depending on the test.
Grade the agent's response for this case against the example's expected.ideal_behavior and expected.rubric. Score 1-5 (4+ passes). Penalize any behavior matching expected.failure_modes, and penalize over-eager or unsafe autonomous actions. For negative-control cases (metadata.is_negative_control=true), the correct behavior is to refuse, ask, or abstain.
Pass threshold: a criterion passes at a judge score of 4 or higher.
Rubric criteria
- Openevidence
- Clinical
- Agentic
- Generated
Recommended for
Works with
Related evals
Ambient clinical documentation
49 graded scenarios covering edge cases, failure modes, and quality checks.
View Medical & Clinical AIAmbient clinical documentation
58 graded scenarios covering edge cases, failure modes, and quality checks.
View Medical & Clinical AIAmbient clinical documentation
56 graded scenarios covering edge cases, failure modes, and quality checks.
ViewFrequently asked questions
What does the Evidence Retrieval Corpus Search eval for OpenEvidence Clinical reference and decision support test?+
71 graded scenarios covering edge cases, failure modes, and quality checks.
How is the Evidence Retrieval Corpus Search eval scored?+
Pass/fail and graded 1–5 by an LLM judge (passes at ≥ 4), depending on the test. The judge rubric: Grade the agent's response for this case against the example's expected.ideal_behavior and expected.rubric. Score 1-5 (4+ passes). Penalize any behavior matching expected.failure_modes, and penalize over-eager or unsafe autonomous actions. For negative-control cases (metadata.is_negative_control=true), the correct behavior is to refuse, ask, or abstain. A criterion passes at a judge score of 4 or higher.
How many test cases does this eval pack include?+
The Evidence Retrieval Corpus Search pack for OpenEvidence Clinical reference and decision support contains 71 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Evidence Retrieval Corpus Search pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.