Eval directory · Accounting & Finance
Evaluation packs covering adversarial robustness, safety gates, workflow quality, and operator-level checks for Puzzle AI products.
Puzzle is an AI-native accounting platform that automates bookkeeping and financial close for startups and growing companies. Its software ingests transactions, reconciles accounts, and surfaces anomalies in real time — reducing close time from weeks to days.
Use the eval library for Puzzle
All 161 test cases — inputs, expected behavior, and pass/fail checks — runnable in Corsac with your own data.
Coverage map
6 areas · 161 graded scenarios
Every eval set is graded on
Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls
01
Operational response/safety eval for Puzzle covering accounting close controls.
Mapped capabilities
12 scenarios
Example criterion: Puzzle enforces accounting close controls with reliable exception detection and action-ready finance guidance.
02
Wave 2 production eval for Puzzle focused on audit readiness traceability.
Mapped capabilities
16 scenarios
Example criterion: Puzzle maintains audit-ready traceability with defensible evidence links and early detection of control gaps.
03
Evaluates Puzzle's Ingest Painpoint Eval — ingest pipeline fault detection, evidence-linked diagnosis, and remediation prioritization — across 12 test cases graded case by case by an LLM judge.
Mapped capabilities
12 scenarios
Example criterion: Puzzle detects ingest painpoints with traceable evidence and prioritized fixes that improve downstream quality.
04
Evaluates Puzzle's Expert Safety Gate Eval V2 High Conf — safety gate enforcement, high-risk scenario handling, and release readiness assurance — across 36 test cases graded case by case by an LLM judge.
Mapped capabilities
36 scenarios
Example criterion: Puzzle enforces expert safety gates reliably and prevents unsafe outputs in production-critical scenarios.
05
Evaluates Puzzle's Power User Ops Eval V2 High Conf — advanced workflow reliability, safety control integrity, and operational consistency — across 40 test cases graded case by case by an LLM judge.
Mapped capabilities
40 scenarios
Example criterion: Puzzle supports power users with reliable operations, strong safety controls, and consistent production-quality execution.
06
Evaluates Puzzle's Workflow Painpoint Eval V2 High Conf — workflow friction detection, severity prioritization, and actionable fix design — across 45 test cases graded case by case by an LLM judge.
Mapped capabilities
45 scenarios
Example criterion: Puzzle consistently surfaces true workflow painpoints, prioritizes them correctly, and provides remediation steps teams can act on.
Each eval pack tests Puzzle's public product surface — including Accounting Close Controls V1, Audit Readiness Traceability V1, Ingest Painpoint Eval V1 — against graded scenarios covering adversarial robustness, workflow quality, safety gates, and operator quality. Every pack is runnable in Corsac with your own data.
Every eval set is graded the same way: pass/fail checks plus an LLM judge scoring 1–5 against each case's expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.
The Puzzle eval library includes 161 graded test cases across 6 eval packs. Each case defines an input, expected behavior, and pass/fail criteria.
Request the library with your work email above and we'll set it up in a Corsac workspace, where you can run every test case against Puzzle or your own agent with your own data.