Eval directory

Evals for Collectly

Eval coverage for Collectly, mapped from its public product surface.

About Collectly

Collectly is an AI-powered revenue cycle management and patient engagement platform for healthcare organizations. Its Billie AI agent runs across the patient financial journey — pre-visit intake, eligibility and benefits verification, cost estimation, point-of-service collection, and post-visit billing follow-up over SMS, email, chat, and voice. It also provides a unified healthcare payments layer with card-on-file, autopay, payment plans, and real-time posting, and integrates with EHR/PM systems such as Epic, Athenahealth, and eClinicalWorks.

Industry

healthcare revenue cycle management (RCM) AI agent & patient payments platform

Website

collectly.co

Use the eval library for Collectly

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Collectly?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Pre-Visit Intake Automation

Digitized registration before arrival: check-in and reminders, insurance capture, data validation, consents, and pre-service payment requests that feed downstream eligibility and billing.

“Digitize check-in, consents, and collect payments before the visit.” www.collectly.co

Mapped capabilities

4 capabilities

Digital check-in and appointment reminders
Pre-visit outreach, form completion, and check-in flow ahead of arrival.
Insurance card capture and coverage status check
Card image capture and real-time coverage status returned before the visit.
Demographic validation and consents
Patient demographic accuracy checks and consent/forms collection.
Pre-service payment requests
Presenting and requesting patient responsibility before the appointment.

AI Eligibility & Benefits Verification

Autonomous retrieval of benefits from clearinghouses and payer portals, plus interpretation of payer responses into visit-specific financial detail with human review only when needed.

Mapped capabilities

4 capabilities

Payer portal and clearinghouse retrieval
Automated benefits lookup across payer access channels.
271 response interpretation
Turning raw payer eligibility responses into readable coverage detail.
Copay, deductible, and responsibility extraction
Surfacing the specific patient-owed amounts for the planned visit.
Human review escalation
Routing ambiguous or incomplete coverage results to staff rather than guessing.

Illustrative example

Input: A payer eligibility response for a scheduled visit shows the plan terminated before the appointment date and returns no active benefit or copay values.
Expected behavior: The agent reports that coverage is inactive as of the termination date and does not present a copay or deductible figure. It flags the record for human review instead of proceeding to a patient responsibility amount.

AI Cost Estimation & Good Faith Estimates

Patient responsibility prediction that combines eligibility data, contracted payer rates, and benefit rules, including automated Good Faith Estimates for No Surprises Act requirements.

“Automated Good Faith Estimates Meet No Surprises Act requirements” www.collectly.co

Mapped capabilities

4 capabilities

Estimate generation from payer data and contracted rates
Applying real rates and benefit rules to planned procedures.
Benefit rule application
Deductible, coinsurance, and copay logic reflected in the estimate.
Automated Good Faith Estimates
Producing GFEs to meet No Surprises Act requirements.
Pre-visit estimate presentation
Communicating the estimate to the patient ahead of care.

Patient Billing Engagement (Billie Agent)

Post-visit balance follow-up handled by the Billie AI agent across SMS, email, chat, and voice — intelligent statements, personalized outreach, billing questions, and payment resolution.

“Collectly's AI-powered revenue layer deploys intelligent agents across every stage of the patient financial journey” www.collectly.co

Mapped capabilities

4 capabilities

Multichannel outreach across SMS, email, chat, and voice
Conducting balance follow-up on the channel the patient uses.
Statement explanation and billing Q&A
Answering repetitive patient billing questions about what they owe and why.
Real-time payment resolution in conversation
Moving a patient from question to completed payment without staff handoff.
24/7 availability and staff handoff
Always-on support with escalation to billing staff when appropriate.

Illustrative example

Input: Patient texts: "Why do I owe $240 when I have insurance?" The account record shows a $240 balance applied to an unmet deductible.
Expected behavior: Billie explains that the $240 was applied to the patient's unmet deductible under their plan, then offers to take payment or set up a payment plan. It gives no clinical or coverage advice beyond the record.

Payments Platform & Revenue Orchestration

A single healthcare-native payments layer spanning pre-visit, point of service, and post-visit — unified rails plus orchestration of payment timing and workflow sequencing.

“Unify pre-visit, point-of-service, post-visit, and AI-driven revenue workflows on a single healthcare-native payments platform.” www.collectly.co

Mapped capabilities

4 capabilities

Card-on-file and tokenization
Securely storing and reusing a patient payment method.
Autopay enrollment and payment plans
Recurring payment setup, plans, and financing options.
Real-time posting and reconciliation
Payments reflected and reconciled against the correct balance in real time.
Intelligent payment timing and sequencing
AI-driven ordering and escalation of payment requests across the lifecycle.

EHR/PM Integration & Data Fidelity

Connectivity into the systems of record named on the site (Epic, Athenahealth, eClinicalWorks, and other listed PM/EHR platforms), so intake, coverage, and payment data land accurately without manual entry.

Mapped capabilities

4 capabilities

Intake and demographic writeback
Delivering captured patient data into the EHR/PM without staff re-keying.
Coverage and balance synchronization
Keeping eligibility results and patient balances consistent with the system of record.
Payment posting into the system of record
Reflecting collected payments back to the practice management ledger.
Cross-system data accuracy
Preventing incomplete or mismatched records that cause downstream claim errors.

Coverage is mapped from Collectly's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Collectly test?+

The coverage map is generated from Collectly's own public product surface (healthcare revenue cycle management (RCM) AI agent & patient payments platform): 6 scoring areas — Pre-Visit Intake Automation, AI Eligibility & Benefits Verification, and AI Cost Estimation & Good Faith Estimates, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Collectly evals scored?+

Every case generated for Collectly — across Pre-Visit Intake Automation and AI Eligibility & Benefits Verification and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Collectly library include?+

The full Collectly library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Digital check-in and appointment reminders and Insurance card capture and coverage status check under Pre-Visit Intake Automation); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Collectly or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Collectly areas and set them up in a Corsac workspace, where you can run every test case against Collectly or your own agent with your own data.