Eval directory

Evals for Numeric

Eval coverage for Numeric, mapped from its public product surface.

About Numeric

Numeric is close-management software for accounting teams that organizes month-end, quarter-end, and year-end tasks while pulling every transaction from the ERP for real-time reconciliation and reporting. Its AI layer ("Numeric Intelligence") handles flux analysis, missing-accrual detection, ad hoc CFO questions, and close post-mortems. A Numeric MCP server connects the workspace to external AI models so agents can run reports and update close tasks.

Industry

AI-powered accounting close and financial reporting automation

Website

www.numeric.io

Use the eval library for Numeric

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Numeric?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Transaction-Grounded Reporting

Building and running reports directly on every transaction-line pulled from the ERP — pivoting, grouping, filtering, and calculating without exporting to Excel or writing a NetSuite Saved Search.

“Numeric pulls every transaction-line from NetSuite so your team can instantly report and search underlying data” www.numeric.io

Mapped capabilities

4 capabilities

Pivot, group, and filter transaction lines
Reports pivoted by dimensions such as department or vendor, with grouping and filters applied to underlying transaction-line data.
Derived metric calculation
Grouping OpEx and revenue accounts and calculating metrics such as operating margin inside the report builder.
Recurring reporting packages
Reports configured to re-run as part of a month-end package and refresh against current GL data.
Natural-language question to report
An ad hoc CFO-style question pasted in and turned into a report by the AI Assistant with no prior setup.

Account Reconciliation

Reconciling balance sheet accounts against ERP transaction data in real time, surfacing discrepancies, and managing the workpapers and auto-submission rules that support them.

“Numeric's data-driven approach to reconciliations gives you completeness in the balance sheet, guaranteed.” www.numeric.io

Mapped capabilities

4 capabilities

Break and discrepancy pinpointing
Identifying the specific GL transactions that caused a previously reconciled account to stop reconciling.
Real-time refresh on new GL activity
Reconciliation state updating as transactions hit the GL, including transactions posting to already-reconciled accounts.
Workpaper connection and editing
Pulling reconciliation balances from cloud storage, data warehouse, or NetSuite Saved Searches and editing workpapers in-platform.
Materiality-based auto-submission
Automatically submitting accounts that tie out and stay under a configured materiality threshold, and withholding those that do not.

Illustrative example

Input: The prepaid insurance account was reconciled and submitted last week. A $4,200 journal entry just posted to it. Show me the current reconciliation status.
Expected behavior: The account is reported as no longer reconciling, with the specific $4,200 GL transaction named as the cause and the resulting difference stated. It is not left in its prior submitted state.

Flux Analysis & Anomaly Detection

Explaining period-over-period movement the way an accountant would, and catching data-quality and completeness problems before they reach the financials.

“Numeric pre-computes your data before your AI model touches it, so you spend less on tokens” www.numeric.io

Mapped capabilities

4 capabilities

Driver-level flux explanation
Explanations that pivot by the relevant ERP dimensions per account rather than restating the variance amount.
Out-of-GL and entity-level context
Incorporating entity-level explanations and payroll or revenue detail sourced outside the general ledger.
Missing accrual detection
Scanning accounts for expenses that appear to lack a corresponding accrual, including via month-over-month comparison.
Tagging and vendor completeness checks
Flagging expenses without a vendor and transactions that appear incorrectly tagged.

Close Orchestration & Controls

Running the close as a controlled project: task checklists across month, quarter, and year end, preparer/reviewer separation, review notes, and the calendar and notification rules around them.

Mapped capabilities

4 capabilities

Task checklist and due-date management
Month-end, quarter-end, and year-end tasks with owners, categories, and status tracked through completion.
Preparer/reviewer segregation of duties
Enforcing that preparation and review are attributed to distinct roles before a task is signed off.
Holiday calendar and due-date shifting
Uploaded holiday calendars affecting when close tasks come due.
Notifications, digests, and request automations
Slack integration, daily digest, and email request automations triggered by close state.

Agentic Access via MCP

The Numeric MCP server exposing the workspace to external AI models: pre-computed report reads plus write-back actions on close tasks that must land where auditors expect them.

“Our MCP syncs changes back to Numeric in real-time, exactly where your auditor expects to find them.” www.numeric.io

Mapped capabilities

4 capabilities

Pre-computed report retrieval
Serving structured, pre-computed data to an external model rather than raw GL transactions.
Task write-back from an AI workspace
Submitting tasks, reassigning owners, and leaving comments from outside Numeric, synced back in real time.
Multi-connector orchestration
Working alongside other connectors so the close checklist guides agent work across sources without manual dataset transfer.
Attribution and audit trail on agent actions
Agent-initiated changes remaining visible and traceable in the close record.

Illustrative example

Input: Via MCP: reassign the 'Accrued bonus rollforward' close task from Dana to Priya and add a comment explaining the handoff.
Expected behavior: The reassignment and comment are applied to that specific task and synced back to Numeric, and the response confirms both actions with the new owner named. No other task is modified.

Traceability & Answer Discipline

The stated posture of 'no black boxes' — outputs that trace back to underlying transactions, stay within what the connected data supports, and remain configurable by the accounting team.

Mapped capabilities

3 capabilities

Drill-down from output to transactions
Any reported figure or explanation traceable to the transaction lines behind it.
Scope discipline on unsupported questions
Declining or qualifying answers that the connected ERP and workpaper data cannot support.
Configurability of AI behavior
Accounting-team control over how flux and analysis steps are configured per account.

Coverage is mapped from Numeric's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Numeric test?+

The coverage map is generated from Numeric's own public product surface (AI-powered accounting close and financial reporting automation): 6 scoring areas — Transaction-Grounded Reporting, Account Reconciliation, and Flux Analysis & Anomaly Detection, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Numeric evals scored?+

Every case generated for Numeric — across Transaction-Grounded Reporting and Account Reconciliation and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Numeric library include?+

The full Numeric library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, Pivot, group, and filter transaction lines and Derived metric calculation under Transaction-Grounded Reporting); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Numeric or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Numeric areas and set them up in a Corsac workspace, where you can run every test case against Numeric or your own agent with your own data.