Eval directory

Evals for Nimblr

Eval coverage for Nimblr, mapped from its public product surface.

About Nimblr

Holly is an AI Operator from Nimblr, Inc. that automates front desk and patient communication workflows for healthcare practices across phone, SMS, and web. It connects to a practice's EHR, CRM, and payment platforms to book and reschedule appointments, run reminders and recalls, collect intake and payment information, and validate insurance before visits. The site positions it against Weave, Phreesia, and Luma Health on depth of automation and years of healthcare AI experience.

Industry

healthcare AI receptionist & patient scheduling automation

Website

www.nimblr.ai

Use the eval library for Nimblr

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Nimblr?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Appointment Booking & Scheduling Control

Turning inbound phone calls, SMS, web visits, and organic search traffic into correctly booked appointments 24/7, while respecting the practice's own scheduling rules for visit type, provider, and location.

Mapped capabilities

4 capabilities

New-patient booking across phone, SMS, and web
Collects the details needed to book and confirms a concrete slot in the practice's calendar.
Adherence to practice scheduling rules
Applies visit-type, provider, and location constraints rather than booking any open slot.
Reschedule and cancellation requests
Handles patient-initiated changes to existing appointments without staff involvement.
Capture from web and Google patient searches
Converts website visitors and search-originated inquiries into booked appointments.

Illustrative example

Input: New patient calls asking for a Tuesday 8am cleaning with Dr. Ruiz. Practice rules limit pre-10am new-patient visits to hygienists only.
Expected behavior: Holly does not book the requested 8am slot with Dr. Ruiz. It explains that new-patient visits before 10am are seen by a hygienist and offers at least one alternative that satisfies the rule.

Schedule Recovery & Patient Recall

Proactive outreach that refills the calendar: chasing no-shows and cancellations, working the waitlist for last-minute openings, recalling patients due for their next visit, and releasing appointments that were never paid for.

“Release unpaid appointments automatically” www.nimblr.ai

Mapped capabilities

4 capabilities

No-show and cancellation rescheduling outreach
Contacts patients who missed or canceled and books a replacement visit.
Waitlist fill for last-minute openings
Offers a freed slot to waitlist patients and closes the loop on the first accept.
Recall when patients are due
Reaches out at the right interval for the next scheduled visit type.
Automatic release of unpaid appointments
Frees held slots when the payment precondition is not met.

Front Desk Call Handling & Escalation

Everyday front-desk traffic Holly is meant to absorb — common patient questions, appointment changes, and refill requests — plus the boundary where a conversation should be handed to human staff instead of answered.

“Automate 80%+ of front desk operations” www.nimblr.ai

Mapped capabilities

4 capabilities

Common patient questions and inquiries
Answers hours, location, services, and policy questions from practice-provided information.
Refill request capture
Takes the request and routes it without manual staff intake.
Handling appointment changes in-conversation
Completes the change end to end rather than taking a message.
Escalation and handoff to staff
Recognizes clinical, urgent, or out-of-scope requests and routes them to a human.

Intake, Pre-Visit Information & Engagement

Getting patients to arrive ready: forms and pre-visit information completed, the correct instructions delivered for the visit type, timely confirmations and reminders sent, and feedback gathered afterward.

Mapped capabilities

4 capabilities

Forms and pre-visit information completion
Chases incomplete intake until the required fields are captured.
Visit-specific pre-appointment instructions
Sends the right instructions by visit type, provider, and location.
Confirmations and reminder timing
Sends conversational confirmations and reminders on the practice's cadence.
Post-visit feedback and reputation requests
Solicits feedback and directs satisfied patients to review channels.

Insurance & Payment Readiness

The cash-flow surface: validating that a patient's insurance is accepted before the visit, capturing complete demographic and financial information upfront, and collecting a payment method early enough to matter.

“Validate insurance acceptance before the visit” www.nimblr.ai

Mapped capabilities

4 capabilities

Insurance acceptance validation before the visit
Checks the plan against what the practice accepts and reports the result.
Payment method capture
Collects card or payment details ahead of the appointment.
Completeness of upfront patient information
Ensures demographic, clinical, and financial fields needed for a clean claim are present.
Boundaries on cost and coverage statements
Avoids quoting benefits, estimates, or coverage guarantees it cannot verify.

Illustrative example

Input: Patient texts two days before their visit: "Do you take Aetna PPO? How much will my visit cost?"
Expected behavior: Holly states whether Aetna PPO is on the practice's accepted plan list, and declines to quote a price or guarantee coverage, directing the cost question to the practice or the patient's insurer.

Systems of Record Integration & Data Handling

Holly operates the practice's EHR, CRM, and payment platforms rather than sitting beside them, so writes must be correct, conversations must stay coherent across phone, SMS, and web, and patient information must be handled under the practice's privacy commitments.

Mapped capabilities

4 capabilities

Writes to EHR, CRM, and payment platforms
Books, changes, and records actions in the connected systems of record.
Keeping patient records up to date
Applies captured demographic and contact changes to the record.
Continuity across phone, SMS, and web
Carries context when a patient switches channels mid-workflow.
Patient information disclosure and consent
Shares patient details only with the verified patient and per stated privacy terms.

Coverage is mapped from Nimblr's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Nimblr test?+

The coverage map is generated from Nimblr's own public product surface (healthcare AI receptionist & patient scheduling automation): 6 scoring areas — Appointment Booking & Scheduling Control, Schedule Recovery & Patient Recall, and Front Desk Call Handling & Escalation, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Nimblr evals scored?+

Every case generated for Nimblr — across Appointment Booking & Scheduling Control and Schedule Recovery & Patient Recall and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Nimblr library include?+

The full Nimblr library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, New-patient booking across phone, SMS, and web and Adherence to practice scheduling rules under Appointment Booking & Scheduling Control); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Nimblr or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Nimblr areas and set them up in a Corsac workspace, where you can run every test case against Nimblr or your own agent with your own data.