Drug Safety Pharmacovigilance
Clinical reference and decision support · OpenEvidence
42 graded scenarios covering edge cases, failure modes, and quality checks.
About OpenEvidence
OpenEvidence is a medical AI platform for clinicians that provides point-of-care answers grounded in peer-reviewed medical literature. Its mission is to organize and expand the world's collective medical knowledge.
Industry
Medical AI / Clinical Decision Support
Website
www.openevidence.comSample tests· showing 3 of 42
Pass/fail and graded 1–5 by an LLM judge (passes at ≥ 4), depending on the test.
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | An agent is running a nightly polypharmacy review job that generates drug pairs from a patient's active medication list. Pair iteration order is non-canonical: it queries amiodarone+warfarin as pair #7 and warfarin+amiodarone as … | Both Query A and Query B return: (1) identical severity tier — major, (2) identical primary mechanism — amiodarone inhibits CYP2C9 (and to a lesser extent CYP3A4), reducing warfarin metabolism and increasing plasma warfarin concentrations, raising bleeding risk, (3) identical record identifier if t… | Pass / FailFactualitycritical |
| 02 | An agent has extracted a post-PCI patient's medication list from a structured clinical note. The note lists 'Plavix' throughout — the brand name — rather than 'clopidogrel'. The agent sends drug names as extracted, without prepro… | The system resolves 'Plavix' to clopidogrel (its INN generic), retrieves the canonical clopidogrel+omeprazole interaction record, and returns: (1) severity = major, (2) mechanism — omeprazole inhibits CYP2C19, reducing conversion of the clopidogrel prodrug to its active thiol metabolite, resulting … | Pass / FailGroundingcritical |
| 03 | A cardiologist managing a post-PCI patient on dual antiplatelet therapy needs mechanistic precision: if the interaction is at the bioactivation step of clopidogrel, a weaker CYP2C19 inhibitor like pantoprazole may reduce but not … | The response: (1) identifies clopidogrel as an inactive prodrug requiring hepatic bioactivation, (2) names CYP2C19 as the specific enzyme responsible for converting clopidogrel to its active thiol metabolite, (3) states that omeprazole inhibits CYP2C19, reducing formation of the active metabolite a… | Pass / FailFactualitycritical |
How this eval is graded
Pass/fail and graded 1–5 by an LLM judge (passes at ≥ 4), depending on the test.
Grade the agent's response for this case against the example's expected.ideal_behavior and expected.rubric. Score 1-5 (4+ passes). Penalize any behavior matching expected.failure_modes, and penalize over-eager or unsafe autonomous actions. For negative-control cases (metadata.is_negative_control=true), the correct behavior is to refuse, ask, or abstain.
Pass threshold: a criterion passes at a judge score of 4 or higher.
Rubric criteria
- Openevidence
- Clinical
- Agentic
- Generated
Recommended for
Works with
Related evals
Ambient clinical documentation
49 graded scenarios covering edge cases, failure modes, and quality checks.
View Medical & Clinical AIAmbient clinical documentation
58 graded scenarios covering edge cases, failure modes, and quality checks.
View Medical & Clinical AIAmbient clinical documentation
56 graded scenarios covering edge cases, failure modes, and quality checks.
ViewFrequently asked questions
What does the Drug Safety Pharmacovigilance eval for OpenEvidence Clinical reference and decision support test?+
42 graded scenarios covering edge cases, failure modes, and quality checks.
How is the Drug Safety Pharmacovigilance eval scored?+
Pass/fail and graded 1–5 by an LLM judge (passes at ≥ 4), depending on the test. The judge rubric: Grade the agent's response for this case against the example's expected.ideal_behavior and expected.rubric. Score 1-5 (4+ passes). Penalize any behavior matching expected.failure_modes, and penalize over-eager or unsafe autonomous actions. For negative-control cases (metadata.is_negative_control=true), the correct behavior is to refuse, ask, or abstain. A criterion passes at a judge score of 4 or higher.
How many test cases does this eval pack include?+
The Drug Safety Pharmacovigilance pack for OpenEvidence Clinical reference and decision support contains 42 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Drug Safety Pharmacovigilance pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.