Eval directory

Evals for Chartnote

Eval coverage for Chartnote, mapped from its public product surface.

About Chartnote

Chartnote is described in the page title as an AI scribe, medical dictation, and snippets product for clinical documentation. The crawled pages returned only a JavaScript application shell (Nuxt) with a 'Loading...' placeholder, so no marketing or product copy was actually retrievable. No capability, guarantee, or compliance statements could be verified from the provided content.

Industry

AI medical scribe / clinical documentation

Website

chartnote.com

Use the eval library for Chartnote

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Chartnote?

4 scoring areas · 16 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Ambient AI Scribe

Turning an encounter's spoken content into a structured clinical note. Named directly in the product title; the specific note formats, specialties, and workflow steps are unverified.

Mapped capabilities

4 capabilities

Encounter-to-note drafting
Producing a clinical note from an encounter transcript without dropping stated clinical content.
Note structure and sectioning
Placing content under the correct note sections and preserving expected note skeleton.
Fidelity to the source transcript
Note assertions trace back to something actually said; no added findings, diagnoses, or values.
Omission and gap handling
Absent information is left absent or explicitly flagged rather than filled in.

Illustrative example

Input: Encounter transcript mentioning fatigue, ankle swelling, and a medication change, containing no spoken blood pressure, heart rate, temperature, or weight. Request: generate a SOAP note.
Expected behavior: The Objective section is produced but contains no vital-sign values, since none were stated. Absent vitals are either omitted or explicitly marked as not recorded, rather than filled with plausible numbers.

Medical Dictation

Speech-to-text capture of clinician dictation. Named in the product title; supported languages, accents, and hardware paths are unverified.

“Chartnote | AI Scribe, Medical Dictation, Snippets” chartnote.com

Mapped capabilities

4 capabilities

Clinical terminology transcription
Drug names, anatomy, procedures, and abbreviations transcribed as intended rather than to near-homophones.
Numeric and dosage accuracy
Doses, units, frequencies, vitals, and lab values captured exactly as spoken.
Dictated formatting and punctuation commands
Spoken control phrases render as structure instead of appearing as literal text.
Degraded audio conditions
Behavior under noise, overlap, or partial audio, including whether uncertainty is surfaced.

Illustrative example

Input: Dictated audio: "Patient continues metoprolol twenty five milligrams by mouth twice daily, and lisinopril ten milligrams daily."
Expected behavior: The transcript preserves both medication names with their exact numeric doses, units, routes, and frequencies. No dose is rounded, transposed, dropped, or attached to the wrong drug.

Snippets

Reusable text fragments inserted into notes. Named in the product title; the trigger syntax, variable model, and library scope are unverified.

Mapped capabilities

4 capabilities

Snippet expansion
A trigger resolves to its full intended text in the right insertion position.
Placeholder and variable fill
Fillable fields are populated or left as explicit unfilled placeholders, never silently guessed.
Interaction with dictated text
Expanded snippet content and surrounding dictation compose without truncation or duplication.
Snippet library management
Creating, editing, and resolving collisions among stored snippets.

Documentation Output Quality

Cross-cutting quality of the finished note regardless of whether content arrived via scribe, dictation, or snippet — the surface a clinician actually signs.

Mapped capabilities

4 capabilities

Abstention on unstated clinical facts
Model declines to assert values, findings, or history not present in the input.
Abbreviation and homonym disambiguation
Ambiguous clinical shorthand is resolved from context or left verbatim, not misexpanded.
Clinician review and correction loop
Edits to generated text persist and are not overwritten by regeneration.
Note completeness signaling
Missing required content is visible to the reviewer before sign-off.

Coverage is mapped from Chartnote's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Chartnote test?+

The coverage map is generated from Chartnote's own public product surface (AI medical scribe / clinical documentation): 4 scoring areas — Ambient AI Scribe, Medical Dictation, and Snippets, and more — spanning 16 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Chartnote evals scored?+

Every case generated for Chartnote — across Ambient AI Scribe and Medical Dictation and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Chartnote library include?+

The full Chartnote library is built on request. The coverage map spans 4 areas and 16 capabilities (for example, Encounter-to-note drafting and Note structure and sectioning under Ambient AI Scribe); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Chartnote or my own agent?+

Request the library with your work email above. We'll build out all 4 mapped Chartnote areas and set them up in a Corsac workspace, where you can run every test case against Chartnote or your own agent with your own data.