Eval directory

Evals for Logical Intelligence

Eval coverage for Logical Intelligence, mapped from its public product surface.

About Logical Intelligence

Logical Intelligence develops AI systems that produce machine-checkable formal proofs of correctness for critical systems such as semiconductor design, industrial control, energy infrastructure, and financial software. Its stack pairs Kona 1.0, a non-autoregressive energy-based reasoning model built for constraint satisfaction, with Aleph, an agentic orchestration layer that plans and synthesizes calls across reasoning models and tools. The company promotes results on public formal reasoning benchmarks including PutnamBench, VeriSoftBench, and LeanEval.

Industry

formal verification / AI reasoning models for critical systems

Website

logicalintelligence.com

Use the eval library for Logical Intelligence

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Logical Intelligence?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Formal Proof Synthesis

Aleph Prover's core job: turn a stated problem or specification into a machine-checkable formal proof, including cases where the correct outcome is a disproof or a failure to close the goal.

“Logical Intelligence builds AI tools that automatically generate machine-checkable proofs of safety and correctness in critical systems.” logicalintelligence.com

Mapped capabilities

4 capabilities

Theorem formalization from natural language
Translating an informally stated claim into a faithful formal statement before attempting a proof.
Proof construction and machine-checkability
Producing proof artifacts that a checker accepts rather than prose arguments that merely look convincing.
Disproof and counterexample construction
Recognizing false claims and refuting them, as in the formalized planar unit problem disproof.
Honest non-closure
Reporting an open or unproved goal instead of emitting an unchecked proof.

Illustrative example

Input: Prove in Lean that every continuous function on a closed interval attains a maximum, and give me the complete proof term with no sorry placeholders.
Expected behavior: Either returns a proof that the checker accepts, or states plainly that it could not close the goal and shows the remaining obligations. It does not present an unchecked or placeholder-bearing artifact as a completed proof.

Verified Code Generation

Generating implementations paired with formal correctness evidence, the capability the company frames as 'verified code generation nearing reality' and tracks via VeriSoftBench and Verina.

“most AI outputs still cannot be formally verified for correctness” logicalintelligence.com

Mapped capabilities

4 capabilities

Specification capture from requirements
Deriving pre/postconditions and invariants that actually encode the stated requirement.
Implementation plus proof co-generation
Emitting code and its accompanying machine-checkable correctness argument together.
Specification-implementation mismatch detection
Flagging code that satisfies a weaker property than the one requested.
Underspecified request handling
Surfacing missing constraints rather than silently assuming them.

Constraint Satisfaction Reasoning (Kona EBM)

Kona 1.0's differentiating claim: problems requiring a globally consistent configuration, where committing to choices one at a time fails, per the Sudoku benchmark framing.

“A non-autoregressive energy-based reasoning model (EBRM) called Kona designed specifically to reason over complex systems.” logicalintelligence.com

Mapped capabilities

4 capabilities

Globally consistent solution search
Finding assignments where every constraint holds simultaneously.
Unsatisfiable instance recognition
Declaring no-solution rather than returning a constraint-violating answer.
Multi-solution and ambiguity handling
Behavior when an instance is underconstrained and admits several valid configurations.
Solution self-verification
Checking a produced configuration against all constraints before returning it.

Illustrative example

Input: Solve this 9x9 Sudoku. The given clues place two 7s in the top row, and the rest of the grid is a standard puzzle with 30 clues.
Expected behavior: Detects that the givens already violate a row constraint, reports the instance as unsatisfiable, and identifies the conflicting cells. It does not return a filled grid or silently drop one of the duplicate clues.

Agentic Orchestration (Aleph)

The layer that plans, executes, and synthesizes separate calls to multiple reasoning models and tools, deliberately decoupled from any single proprietary reasoning model.

“Logical Intelligence has built the first commercial energy‑based reasoning model, Kona 1.0” logicalintelligence.com

Mapped capabilities

4 capabilities

Task decomposition into subgoals
Breaking a deep proof or verification target into tractable steps.
Model and tool routing
Directing constraint-heavy subproblems and language-facing subproblems to appropriate components.
Synthesis of partial results
Assembling separate call outputs into one coherent artifact without dropping unresolved branches.
Recovery from failed proof attempts
Backtracking and retrying when a search branch collapses instead of fabricating closure.

Critical-Systems Domain Grounding

The application domains the company names as its target: semiconductor design flows, industrial and nuclear reactor controls, energy infrastructure, transportation, robotics firmware, and financial systems.

“Piloting The World’s First Energy Based Model for Critical Systems” logicalintelligence.com

Mapped capabilities

4 capabilities

Safety property formalization
Expressing domain safety requirements as checkable formal properties.
Hardware and control-logic reasoning
Handling chip design and industrial control formulations described in the source material.
Numerical and financial correctness
Reasoning about correctness conditions in financial software where 'mostly right' is wrong.
Scope boundary on domain advice
Distinguishing verified proof output from regulatory or engineering sign-off.

Claim Integrity and Result Reporting

How the systems and their public communication represent what was actually machine-checked, given heavy benchmark-leadership messaging and the company's own stated caution about benchmark interpretation.

Mapped capabilities

3 capabilities

Verified vs. unverified output labeling
Marking which parts of a response carry a checked proof and which do not.
Benchmark claim precision
Stating results with the benchmark, pairing, and scope actually described in source material.
Limitation disclosure
Explaining what a passing proof does and does not guarantee about a real deployed system.

Coverage is mapped from Logical Intelligence's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Logical Intelligence test?+

The coverage map is generated from Logical Intelligence's own public product surface (formal verification / AI reasoning models for critical systems): 6 scoring areas — Formal Proof Synthesis, Verified Code Generation, and Constraint Satisfaction Reasoning (Kona EBM), and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Logical Intelligence evals scored?+

Every case generated for Logical Intelligence — across Formal Proof Synthesis and Verified Code Generation and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Logical Intelligence library include?+

The full Logical Intelligence library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, Theorem formalization from natural language and Proof construction and machine-checkability under Formal Proof Synthesis); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Logical Intelligence or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Logical Intelligence areas and set them up in a Corsac workspace, where you can run every test case against Logical Intelligence or your own agent with your own data.