Eval directory

Evals for Pulse AI

Eval coverage for Pulse AI, mapped from its public product surface.

About Pulse AI

Pulse is a document intelligence platform that combines OCR, layout detection, and vision models to turn complex unstructured documents into structured outputs such as JSON, tables, and bounding boxes. It targets enterprises and AI-native teams, with a multi-stage pipeline covering layout understanding, component-specific OCR, reading order, table structure recognition, and fine-tuned VLMs for charts and figures. It offers self-serve and enterprise tiers, flexible deployment (VPC, on-prem, air-gapped), and recent additions including document metadata extraction, document classification, and agentic chart reconstruction.

Industry

enterprise document processing and OCR API

Website

www.runpulse.com

Use the eval library for Pulse AI

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Pulse AI?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Layout Understanding & Reading Order

Page structure is resolved before content is read: components are detected and classified, and text is emitted in human-like sequence across irregular layouts.

“PulseBench-Tab is a multilingual benchmark with 1,820 human-annotated tables across 9 languages” www.runpulse.com

Mapped capabilities

4 capabilities

Component detection and typing
Distinguishing tables, charts, figures, headers, and body text regions on a page.
Multi-column and irregular layout sequencing
Reading order across columns, sidebars, and non-linear page designs.
Cross-page continuity
Sections, headers, and tables that span page boundaries in multi-page documents.
Layout-preserving hierarchy
Retaining indentation and nesting that carry meaning, as in financial statements.

Table Structure Recognition & Parsing

Recovery of true table structure — not just cell text — including spans, headers, and adjacency, across languages and degraded scans, in line with the T-LAG framing of PulseBench-Tab.

Mapped capabilities

4 capabilities

Rowspan/colspan and header structure
Merged cells, thead/tbody separation, and nested header rows.
Degraded-input robustness
Low-fidelity scans, rotations, photocopies, and handwritten table content.
Multilingual and multi-script tables
Latin, CJK, Arabic, and Cyrillic script tables.
Large and dense tables
High cell-count tables where structural relationships must survive at scale.

Illustrative example

Input: A scanned, slightly rotated balance sheet page whose table has a two-row header with one header cell spanning three quarterly columns, plus an indented three-level asset breakdown.
Expected behavior: The returned table reproduces the two-row header with the spanning cell marked as covering three columns, keeps every data row aligned to the correct quarter, and preserves the nesting depth of the indented asset lines.

Chart & Figure Reconstruction

Agentic reconstruction of the data encoded visually in charts — values, scales, series, and encodings — rather than only the surrounding text, with linkage back to the source document.

Mapped capabilities

4 capabilities

Value and scale recovery
Bar heights, line slopes, point positions, and axis scale inference.
Series and legend attribution
Assigning recovered values to the correct series across overlapping curves.
Chart-type coverage
Line, scatter, bar, pie, donut, and well-log charts.
Source-document grounding
Preserving the connection between reconstructed data and its location on the page.

Structured Output & Schema Conformance

The contract Pulse returns to downstream systems: valid JSON, user-defined schemas, bounding-box coordinates, table and image summarization, and native file metadata in the same Extract call.

Mapped capabilities

4 capabilities

User-defined schema adherence
Output conforms to a caller-supplied JSON schema without invented fields.
Bounding-box coordinate fidelity
Coordinates that correctly localize extracted content on the page.
Document metadata extraction
extensions.document_metadata: file identity, hashes, authorship, timestamps, hidden sheets, EXIF, deterministic structure.
Absent-field handling
Omitting fields that do not exist in the source rather than emitting nulls or guesses.

Illustrative example

Input: An Extract call on a ten-page PDF with extensions.document_metadata set to true, where the file has a title and two authors but no keywords property.
Expected behavior: The response returns extracted content unchanged and adds extensions.document_metadata containing file identity, byte size, media type, SHA-256 hash, title, and both authors. The keywords field is omitted entirely rather than returned as null or an empty value.

Classification & Intake Routing

Classify identifies document type before extraction and routes each file into the appropriate pipeline, removing manual triage at intake.

Mapped capabilities

4 capabilities

Document type identification
Correct type assignment across mixed enterprise intake batches.
Pipeline routing decisions
Selecting the extraction path implied by the detected type.
Mixed-format intake
PDFs, images, spreadsheets, and Office files arriving in one stream.
Ambiguous or unrecognized documents
Behavior when type confidence is low or the document does not match a known class.

Delivery, Deployment & Data Handling

Operational surfaces around the API: asynchronous delivery, access modes, and the deployment and data-residency commitments that gate enterprise adoption.

Mapped capabilities

4 capabilities

Webhook callback delivery
Asynchronous completion callbacks for long-running or batched extraction.
Agentic payment access path
Per-call payment by an agent with no account, API key, or signup.
Zero-data-retention and residency
Retention, US/EU residency, and BYOK behavior on enterprise tiers.
Isolated deployment modes
VPC, on-prem, and air-gapped operation with no external dependencies.

Coverage is mapped from Pulse AI's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Pulse AI test?+

The coverage map is generated from Pulse AI's own public product surface (enterprise document processing and OCR API): 6 scoring areas — Layout Understanding & Reading Order, Table Structure Recognition & Parsing, and Chart & Figure Reconstruction, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Pulse AI evals scored?+

Every case generated for Pulse AI — across Layout Understanding & Reading Order and Table Structure Recognition & Parsing and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Pulse AI library include?+

The full Pulse AI library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Component detection and typing and Multi-column and irregular layout sequencing under Layout Understanding & Reading Order); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Pulse AI or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Pulse AI areas and set them up in a Corsac workspace, where you can run every test case against Pulse AI or your own agent with your own data.