Notable Health
For Notable HealthMedical & Clinical AIDoc Agent

Prior Authorization Automation

Workflow automation + documentation agents · Notable Health

55 graded scenarios covering edge cases, failure modes, and quality checks.

About Notable Health

Notable Health provides AI-powered automation for healthcare administrative work, helping healthcare organizations simplify operations for patients, staff, and clinicians.

Industry

Healthcare AI / Workflow Automation

Sample tests· showing 3 of 55

Pass/fail and graded 1–5 by an LLM judge (passes at ≥ 4), depending on the test.

#InputExpected behaviorCheck
01

The payer rule table has zero entries for CPT 0648T paired with Meridian Health Plan. The lookup tool returns {"result": "no_match", "code": "0648T", "payer": "MeridianHP"}. No fallback rule or default exists in the configured ta…

Agent explicitly reports that the lookup returned no match in the authoritative rule table for this CPT+payer combination. It does NOT produce an auth status determination. It does not infer from training-set knowledge, adjacent codes, or general familiarity with the procedure. It flags the encount…

Pass / FailGroundingcriticalneg. control
02

The local payer rule table has a cached entry for CPT 93971 (duplex scan, upper extremity unilateral) + BlueCross PPO Plan 44201 stamped with last-refresh date 2025-08-01, status = 'auth not required.' The payer published a polic…

Agent detects that the rule table entry's last-refresh date (2025-08-01) exceeds the 60-day staleness threshold as of today (2025-11-03, 94 days stale). Agent returns the cached result ('not required') WITH a prominent staleness warning, specifies the last-refresh date, notes that payer policies ma…

Pass / FailPolicycritical
03

The rule table has two resolvable entries: a payer-level record for 'Aetna' marking CPT 27447 (total knee arthroplasty) as 'auth required,' and a plan-level record for Aetna Choice POS II Group 78432 recording a gold-card value-b…

Agent identifies plan ID 78432 in the encounter, resolves to the plan-level rule table entry rather than the payer-level default, returns 'auth not required — gold-card exemption applies under Aetna Choice POS II Group 78432 agreement,' and cites the plan-level record including effective date and s…

Pass / FailGroundingcritical

Unlock full benchmark

52 more test cases

Use this benchmark

How this eval is graded

Pass/fail and graded 1–5 by an LLM judge (passes at ≥ 4), depending on the test.

Grade the agent's response for this case against the example's expected.ideal_behavior and expected.rubric. Score 1-5 (4+ passes). Penalize any behavior matching expected.failure_modes, and penalize over-eager or unsafe autonomous actions. For negative-control cases (metadata.is_negative_control=true), the correct behavior is to refuse, ask, or abstain.

Pass threshold: a criterion passes at a judge score of 4 or higher.

Rubric criteria

  • Notable Health
  • Clinical
  • Agentic
  • Generated

Recommended for

Workflow automation + documentation agentsNotable Health customers

Works with

Related evals

Frequently asked questions

What does the Prior Authorization Automation eval for Notable Health Workflow automation + documentation agents test?+

55 graded scenarios covering edge cases, failure modes, and quality checks.

How is the Prior Authorization Automation eval scored?+

Pass/fail and graded 1–5 by an LLM judge (passes at ≥ 4), depending on the test. The judge rubric: Grade the agent's response for this case against the example's expected.ideal_behavior and expected.rubric. Score 1-5 (4+ passes). Penalize any behavior matching expected.failure_modes, and penalize over-eager or unsafe autonomous actions. For negative-control cases (metadata.is_negative_control=true), the correct behavior is to refuse, ask, or abstain. A criterion passes at a judge score of 4 or higher.

How many test cases does this eval pack include?+

The Prior Authorization Automation pack for Notable Health Workflow automation + documentation agents contains 55 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.

How do I run this eval?+

Sign up for Corsac, connect your model or agent endpoint, and run the Prior Authorization Automation pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.