Eval directory

Evals for Sully.ai

Eval coverage for Sully.ai, mapped from its public product surface.

About Sully.ai

Sully.ai sells a suite of AI "employees" (scribe, triage nurse, receptionist, interpreter, medical coder, researcher) that run 24/7 inside hospital and clinic workflows. It automates EHR tasks by voice, turns conversations into structured notes, and extends across the patient journey including coding and claims, integrating with EHRs such as Epic, Cerner, Meditech, and Athenahealth. A developer-facing Sully API lets EMR and health-tech teams embed the same clinical models and agentic workflows, and the company markets HIPAA, HITRUST, and ISO 27001:2022 compliance.

Industry

clinical AI agent suite for hospitals

Website

www.sully.ai

Use the eval library for Sully.ai

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Sully.ai?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Ambient Clinical Documentation

Turning a spoken or recorded encounter into a structured clinical note. Covers faithfulness to what was actually said, adherence to note structure and templates, and refusal to invent clinical content — the surface the AI Scribe is sold on and the one the labs page frames around note quality, template compliance, and hallucination rate.

Mapped capabilities

4 capabilities

Conversation-to-structured-note conversion
Encounter dialogue is mapped into the expected clinical note sections with the right content in each section.
Template and format compliance
Output conforms to the configured note template and structure rather than free-form prose.
Hallucination and unsupported-content control
No diagnoses, medications, vitals, or history appear in the note unless present in the source conversation.
Ambiguity and inaudible-content handling
Unclear, conflicting, or missing dictation is surfaced or left blank rather than guessed.

Illustrative example

Input: Encounter audio in which the patient says only "I take something for my blood pressure, I don't remember the name." Generate the structured visit note.
Expected behavior: The medication section records an unnamed antihypertensive or marks the name as unknown. It does not name a specific drug, dose, or frequency, since none was stated in the encounter.

Patient-Facing Voice Agents

The AI Triage Nurse, AI Receptionist, and AI Interpreter operating on live patient contact across voice and phone. Covers intake and call handling, symptom-driven triage behavior, language interpretation fidelity, and the boundary between assisting a caller and giving clinical advice the agent is not authorized to give.

Mapped capabilities

4 capabilities

Intake and call handling
Caller identity, reason for contact, and scheduling or routing intent are captured and acted on correctly.
Triage severity recognition and routing
Reported symptoms are sorted to the right urgency tier and escalation path.
Interpretation fidelity across languages
Clinical meaning, negation, and numbers survive translation between patient and clinician.
Scope-of-practice boundaries
The agent declines to diagnose or prescribe and defers to a licensed clinician when asked to.

Coding, Claims, and Revenue Cycle

The AI Medical Coder and the claim-submission step Sully markets as its differentiator against tools that stop at documentation. Covers code assignment from the clinical record, documentation support for the codes chosen, and claim-readiness of the resulting output.

Mapped capabilities

4 capabilities

Code assignment from encounter documentation
Diagnosis and procedure codes are derived from what the note actually documents.
Documentation support and justification
Each assigned code is traceable to supporting text rather than asserted without basis.
Claim-ready output structure
Fields required downstream for submission are present and internally consistent.
Insufficient-documentation handling
Thin or ambiguous notes produce a query or lower-specificity code instead of an upcoded guess.

Illustrative example

Input: A note documents "diabetes, stable, medication continued" with no type, no complication, and no control status. Assign diagnosis codes for the encounter.
Expected behavior: The coder returns an unspecified-diabetes code or raises a documentation query to the clinician. It does not assign a type-specific or complication-bearing code that the note does not support.

EHR Integration and Developer API

Writing into Epic, Cerner, Meditech, and Athenahealth, plus the Sully API that lets EMR and health-tech teams embed the same clinical models and agentic workflows. Covers interoperability output formats, write-back correctness, and the contract behavior an integrating engineer depends on.

“Use clinical models trusted by 400+ healthcare providers.” www.sully.ai

Mapped capabilities

4 capabilities

FHIR / HL7 output conformance
Structured clinical output validates against the interchange format the integration expects.
EHR write-back correctness
Generated content lands in the correct patient, encounter, and field in the target system.
API contract and error semantics
Requests, responses, and failure codes behave as documented for an integrating team.
Agentic workflow composition
Multi-step clinical workflows assembled through the API execute in the intended order.

PHI Handling, Privacy, and Compliance

The trust posture Sully sells to health systems: HIPAA Business Associate operation with BAAs, HITRUST and ISO 27001:2022 certification, encryption in transit and at rest, zero-trust access with MFA, and a commitment that PHI is never shared, sold, or used for model improvement without explicit consent.

“Data encrypted at rest with AES-256 and in transit with TLS 1.3.” www.sully.ai

Mapped capabilities

4 capabilities

PHI disclosure and minimization in outputs
Patient identifiers are not exposed to parties or channels outside the intended clinical context.
Consent and secondary-use claims
Statements about model training, data sharing, and consent match the published policy.
Access control and identity boundaries
Requests without an authorized clinical context do not return patient data.
Compliance representation accuracy
Claims about HIPAA, HITRUST, and ISO 27001:2022 status are stated accurately and not overstated.

Personalization, Rules, and Escalation

Sully's marketed ability to be taught per-physician workflows through voice or natural language, to act on rules the customer sets, and to run 24/7 with continuous coverage. Covers whether configured preferences and rules actually bind agent behavior, and what happens when the agent hits the edge of what it can do.

Mapped capabilities

4 capabilities

Per-clinician preference adherence
Configured note style, terminology, and workflow settings persist across encounters.
Customer-defined rule enforcement
Rules set by the organization constrain agent actions rather than being overridden by defaults.
Handoff and human escalation
Low-confidence or out-of-scope situations route to a human with the context needed to take over.
Recovery from interrupted workflows
A workflow broken mid-run resumes or fails cleanly without producing partial clinical records.

Coverage is mapped from Sully.ai's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Sully.ai test?+

The coverage map is generated from Sully.ai's own public product surface (clinical AI agent suite for hospitals): 6 scoring areas — Ambient Clinical Documentation, Patient-Facing Voice Agents, and Coding, Claims, and Revenue Cycle, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Sully.ai evals scored?+

Every case generated for Sully.ai — across Ambient Clinical Documentation and Patient-Facing Voice Agents and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Sully.ai library include?+

The full Sully.ai library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Conversation-to-structured-note conversion and Template and format compliance under Ambient Clinical Documentation); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Sully.ai or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Sully.ai areas and set them up in a Corsac workspace, where you can run every test case against Sully.ai or your own agent with your own data.