Eval directory

Evals for Liquid AI

Eval coverage for Liquid AI, mapped from its public product surface.

About Liquid AI

Liquid AI builds Liquid Foundation Models (LFMs), a family of small device-native foundation models designed to run on phones, laptops, cars and other hardware outside the data center. The lineup spans LFM2.5-230M through the 24B-parameter sparse MoE LFM2-24B-A2B, with base and post-trained checkpoints distributed on Hugging Face and support for runtimes such as llama.cpp, MLX, ONNX, CoreML, vLLM and SGLang. The models target agentic workloads — tool calling, planning, structured output and multi-step tasks — with local inference pitched as low-latency, private and free at the margin.

Industry

on-device / edge foundation models

Website

www.liquid.ai

Use the eval library for Liquid AI

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Liquid AI?

6 scoring areas · 22 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Agentic tool calling and multi-step execution

The headline workload for LFM2.5-2.6B and LFM2.5-8B-A1B: selecting tools, binding arguments correctly, chaining calls, and driving a multi-step task to completion on-device.

“an agentic model that runs entirely on-device” www.liquid.ai

Mapped capabilities

4 capabilities

Tool selection and argument binding
Choosing the right tool from a provided set and populating required parameters with well-typed values drawn from the user request.
Sequential tool chaining
Ordering dependent calls so that later calls consume earlier results, rather than emitting them in parallel or out of order.
Instruction-to-skill decomposition
Breaking a single free-form command into an ordered sequence of discrete skill or tool invocations, as in the robot skill-selection deployment.
Termination and result grounding
Stopping once the task is satisfied and reporting an answer grounded in returned tool output instead of continuing to call tools.

Illustrative example

Input: Tools: get_calendar(date), send_message(contact, text). User: "Check whether I'm free Tuesday afternoon, and if I am, message Priya that I can make the 3pm."
Expected behavior: The model calls get_calendar for Tuesday first and waits for the result. It issues send_message to Priya only if the returned schedule shows no conflict at 3pm; otherwise it reports the conflict and sends nothing.

Structured output and schema compliance

The surface IFStruct was built to measure: producing valid, parseable output that satisfies the exact fields, types, enums and formatting the user asked for.

Mapped capabilities

4 capabilities

Field, type and enum adherence
Emitting exactly the requested keys with correct types and values restricted to the specified enumerations.
Escaping and nesting correctness
Producing syntactically valid output when strings require escaping or the schema nests objects and arrays.
Varied schema phrasings
Honoring schema requirements whether they arrive as a formal spec, an inline description, or prose instructions.
Output-only discipline
Returning the structure alone, without preamble, commentary, or code-fence wrappers, when asked for raw structured output.

Illustrative example

Input: Return two blueberry pancake recipes as JSON only. Each object needs name (string), minutes (integer), and difficulty, which must be exactly one of: easy, medium, hard.
Expected behavior: The response is a bare JSON array of exactly two objects. Each carries only the three requested keys, with name a string, minutes an integer, and difficulty drawn from the given enum. No prose, preamble, or code fence surrounds it.

Instruction following, extraction and multilingual input

General-purpose competence the models are pitched on beyond tool use: multi-constraint instructions, pulling fields out of unstructured text, and handling non-Latin scripts after the vocabulary expansion to 128K.

Mapped capabilities

4 capabilities

Multi-constraint instruction following
Satisfying several simultaneous constraints on content, length and format within one response.
Data extraction from unstructured text
Locating and normalizing requested values from messy source text, a task called out as a strength even at 230M.
Abstention over fabrication
Declining to answer when the requested fact is absent or unknown, the behavior the AA-Omniscience framing penalizes hallucination on.
Non-Latin script handling
Comparable instruction following and output quality in scripts targeted by the tokenizer expansion, such as Hindi, Thai, Vietnamese, Indonesian and Arabic.

Long-context behavior

How the models hold up across the 32K and 128K context windows introduced through dedicated context-extension phases, which agentic workloads depend on.

Mapped capabilities

3 capabilities

Retrieval from long inputs
Finding and using specific content placed deep inside a document near the top of the supported window.
Long agentic trace coherence
Maintaining task state and prior tool results across an extended multi-turn trajectory.
Context-tier consistency
Behaving consistently at the window each checkpoint actually advertises, rather than degrading silently past its limit.

Generation stability and failure recovery

Documented failure modes and how the model gets out of them — most explicitly the repetitive 'doom loop' that Antidoom and FTPO were built to reduce.

“After Antidoom training, that rate fell to 1.4%” www.liquid.ai

Mapped capabilities

3 capabilities

Repetition and doom-loop avoidance
Completing hard math and coding prompts without degenerating into a repeated span until the context is exhausted.
Recovery after a failed tool call
Reacting to an error or empty result from a tool by adapting or reporting, rather than retrying the identical call indefinitely.
Long reasoning-trace termination
Bringing an extended thinking trace to a definite answer instead of looping on reconsideration.

Deployment portability and checkpoint parity

The practical surface of shipping a device-native model: consistent behavior across runtimes, across the 230M-to-24B-A2B size ladder, and between base and post-trained checkpoints.

Mapped capabilities

4 capabilities

Cross-runtime behavioral parity
Equivalent responses to the same prompt across llama.cpp, MLX, ONNX, CoreML, vLLM and SGLang builds of one checkpoint.
Size-tier capability ordering
Whether capability holds or improves moving up the ladder from LFM2.5-230M to LFM2-24B-A2B on the same task.
Base versus post-trained behavior
Distinguishing the instruction-following and agentic behavior expected of post-trained checkpoints from the base ones meant for fine-tuning.
Resource-envelope conformance
Operating within the stated footprint, such as LFM2-24B-A2B fitting in 32GB of RAM or sub-1GB on-device checkpoints.

Coverage is mapped from Liquid AI's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Liquid AI test?+

The coverage map is generated from Liquid AI's own public product surface (on-device / edge foundation models): 6 scoring areas — Agentic tool calling and multi-step execution, Structured output and schema compliance, and Instruction following, extraction and multilingual input, and more — spanning 22 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Liquid AI evals scored?+

Every case generated for Liquid AI — across Agentic tool calling and multi-step execution and Structured output and schema compliance and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Liquid AI library include?+

The full Liquid AI library is built on request. The coverage map spans 6 areas and 22 capabilities (for example, Tool selection and argument binding and Sequential tool chaining under Agentic tool calling and multi-step execution); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Liquid AI or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Liquid AI areas and set them up in a Corsac workspace, where you can run every test case against Liquid AI or your own agent with your own data.