Eval directory

Evals for PrismML

Eval coverage for PrismML, mapped from its public product surface.

About PrismML

Prism ML builds ultra-dense, heavily quantized language and image models under the Bonsai brand, offered in 1-bit and Ternary variants at 1.7B, 4B, 8B, and 27B classes. The pages present Bonsai 27B as a 27B-class model small enough to run on a phone, with multi-step reasoning, tool calling, agentic workflows, and multimodal understanding on local devices. Marketing centers on intelligence density — comparable capability at far lower memory, latency, and energy cost — with benchmarks referred to a whitepaper.

Industry

on-device compressed LLMs / edge AI models

Website

prismml.com

Use the eval library for PrismML

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for PrismML?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Multi-Step Reasoning Under Compression

Whether the heavily quantized models hold a chain of reasoning together — the capability the pages claim survives 1-bit and Ternary compression. Focuses on where quantization typically degrades first: intermediate state, arithmetic, and constraint tracking.

“It brings multi-step reasoning, tool calling, agentic workflows, and multimodal understanding to local devices.” prismml.com

Mapped capabilities

4 capabilities

Multi-hop question decomposition
Questions requiring two or more dependent inference steps before a final answer.
Arithmetic and unit consistency
Numeric chains where an early rounding or unit slip propagates to a wrong final value.
Constraint-following across a long instruction
Prompts stacking several explicit format, length, or content constraints at once.
Abstention on underspecified input
Cases where the correct move is to ask or decline rather than to guess a missing premise.

Tool Calling and Agentic Workflows

The pages claim tool calling and agentic workflows run locally. This area covers whether tool invocations are schema-valid, correctly argued from user input, and correctly sequenced — the failure surface that breaks an on-device agent silently.

Mapped capabilities

4 capabilities

Schema-valid invocation
Emitted calls match the declared tool name, required fields, and types.
Argument grounding
Arguments come from the user turn or a prior tool result, not from invention.
Multi-step tool sequencing
Dependent calls ordered so each consumes the prior result rather than running in parallel.
Knowing when not to call
Direct answers when no tool applies, and no speculative results before a tool returns.

Illustrative example

Input: Tools available: get_weather(city, unit) and send_sms(to, body). Text my brother at +1-555-0142 tomorrow's forecast for Pasadena in Celsius.
Expected behavior: The model calls get_weather with city Pasadena and unit Celsius first, waits for the result, then calls send_sms to +1-555-0142 with that forecast in the body. It states no temperature before the weather tool returns.

Multimodal Understanding

Bonsai 27B is presented as multimodal on local devices, and Bonsai Image ships as a separate family. Covers image-grounded response quality and the specific risk that a compressed vision path asserts detail it cannot actually resolve.

Mapped capabilities

4 capabilities

Image content question answering
Questions answerable strictly from what is visible in the supplied image.
Text and figure reading in images
Reading labels, screenshots, or chart values rendered inside an image.
Image-grounded restraint
Declining to assert objects, text, or counts not present or not legible.
Image plus instruction tasks
Combining a visual input with a formatting or transformation instruction.

Variant and Size-Class Selection

The product's core decision for a buyer is which of eight published configurations to ship. Covers whether the surface routes a user to the right variant and size class given a device and footprint budget, using only the sizes the pages state.

Mapped capabilities

4 capabilities

1-bit versus Ternary tradeoff
Explaining the footprint-versus-fidelity difference between the two quantizations.
Device fit routing
Phone targets to 1-bit 27B at 3.9GB; laptop targets to Ternary 27B at 5.9GB.
Size-class routing
Choosing among 1.7B, 4B, 8B, and 27B classes for a stated workload.
Download and docs handoff
Pointing to the correct download, demo, or docs destination for the chosen variant.

Illustrative example

Input: I'm shipping an iPhone app with about a 4GB budget for the model file. Which Bonsai 27B variant fits, and how big is it?
Expected behavior: The response recommends 1-bit Bonsai 27B and gives its size as 3.9GB. It notes that Ternary Bonsai 27B is 5.9GB and targeted at laptops, so it does not fit the stated budget.

Performance Claim Fidelity

Headline efficiency claims (14x less memory, 8x faster, 5x less energy) and all benchmark charts are deferred to a whitepaper not reproduced on the site. Covers whether the surface restates these claims accurately and scoped, without inventing scores.

“14 × less memory 8 × faster 5 × less energy” prismml.com

Mapped capabilities

4 capabilities

Headline multiplier accuracy
Restating the memory, speed, and energy multipliers exactly as published.
Comparison baseline disclosure
Naming what a multiplier is measured against rather than presenting it as absolute.
Intelligence-density definition
Describing the metric as published: negative log error rate divided by model size.
No fabricated benchmark numbers
Deferring to the whitepaper instead of producing per-benchmark scores or rankings.

Site Intake and Failure States

The public site's conversion paths — role applications with resume upload, contact, and newsletter signup — each expose explicit success and error states in the markup. Covers whether those paths validate, confirm, and recover legibly.

Mapped capabilities

4 capabilities

Role application submission
Selecting a listed role and completing the application to a confirmed state.
Resume upload limits
Files over the stated 10MB maximum produce the failure state, not a silent drop.
Contact and newsletter confirmation
Successful submission surfaces the confirmation rather than leaving the form ambiguous.
Error-state recovery
After a failed submission, entered input is recoverable and the retry path is clear.

Coverage is mapped from PrismML's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for PrismML test?+

The coverage map is generated from PrismML's own public product surface (on-device compressed LLMs / edge AI models): 6 scoring areas — Multi-Step Reasoning Under Compression, Tool Calling and Agentic Workflows, and Multimodal Understanding, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the PrismML evals scored?+

Every case generated for PrismML — across Multi-Step Reasoning Under Compression and Tool Calling and Agentic Workflows and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the PrismML library include?+

The full PrismML library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Multi-hop question decomposition and Arithmetic and unit consistency under Multi-Step Reasoning Under Compression); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against PrismML or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped PrismML areas and set them up in a Corsac workspace, where you can run every test case against PrismML or your own agent with your own data.