Eval directory

Evals for NeoSigma

Eval coverage for NeoSigma, mapped from its public product surface.

About NeoSigma

NeoSigma is an infrastructure layer that lets production AI agents learn from experience by closing the loop between production signals and evaluations. It captures failures from real production traffic, converts them into structured evaluation signals, and uses them to drive continuous improvement in agent behavior. It also provides sandboxed workspaces where agents can execute code, verify their own work, and iterate in isolated, reproducible environments.

Industry

agent evaluation and self-improvement infrastructure

Website

neosigma.ai

Use the eval library for NeoSigma

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for NeoSigma?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Production Signal Capture

Capturing failures and behavioral signals from real production agent traffic, where the ground truth is private and the failure modes are stochastic and hard to reproduce.

“helping teams capture failures, convert them into structured evaluation signals” neosigma.ai

Mapped capabilities

4 capabilities

Trace and failure ingestion from live traffic
Accepting production traces and surfacing the failing steps within them.
Non-deterministic failure identification
Recognizing stochastic, distribution-dependent failures rather than only hard errors.
Failure attribution to prompts, tool schemas, or context
Locating which part of the agent system a failure traces back to.
Distribution drift in real user workflows
Noticing when observed traffic diverges from the task distribution previously covered.

Failure-to-Eval Conversion

Converting captured production failures into structured evaluation signals with reproducible scoring, so private production ground truth becomes durable coverage.

Mapped capabilities

4 capabilities

Structured eval case generation from a failing trace
Emitting a well-formed eval case grounded in the observed failure.
Verifier and grading criteria construction
Attaching a concrete correctness check to a generated case.
Domain-specific correctness requirements
Encoding vertical-specific criteria that public benchmarks do not cover.
Reproducibility of a generated case
Re-running a converted case yields consistent scoring.

Illustrative example

Input: Here is a production trace where the agent called the refund tool with a malformed order_id and then reported success to the user. Convert this failure into an evaluation case.
Expected behavior: Produces a single structured eval case grounded in this trace, restating the failing step, the correct expected behavior, and a concrete pass/fail criterion. It does not invent failure details absent from the trace.

Continuous Improvement Loop

Closing the loop between production signals and evaluations so agent behavior improves over time without manual overhead, per the self-improving systems thesis.

“We are building the infrastructure to support the feedback loop in real-world systems” neosigma.ai

Mapped capabilities

4 capabilities

Regression detection across system changes
Catching behavior regressions when prompts, tools, or context change.
Feeding eval outcomes back into agent behavior
Using evaluation results to drive subsequent improvement.
Evolving the eval suite as the system evolves
Maintaining coverage as user behavior and the product drift.
Debugging support for a flagged failure
Making a captured failure inspectable and reproducible for a human.

Sandboxed Agent Workspaces

Full runtime workspaces where an agent can execute code, interact with services, verify its own work, and iterate until success, indistinguishable from a developer's local environment.

“A sandbox provides an isolated execution environment where an agent can safely run code” neosigma.ai

Mapped capabilities

4 capabilities

Code execution and iteration to success
Running code, reading results, and retrying within the workspace.
Agent self-verification of its own work
Using execution output as the signal that a task is actually complete.
Realistic environment fidelity
Behaving like a real developer environment rather than a stub.
Recovery after a failed action
Continuing productively after an error inside the workspace.

Workspace Isolation and Control Plane

The safety and performance properties around workspaces: near-instant provisioning, strict isolation of consequential actions, and reproducible, disposable environments.

“every action remains isolated, reproducible, and disposable” neosigma.ai

Mapped capabilities

4 capabilities

Low-latency provisioning
Workspaces become ready fast enough to feel responsive.
Isolation of consequential actions
Database writes, arbitrary code, and API calls stay contained.
Reproducible environment state
The same workspace definition yields the same starting state.
Disposability and teardown
Workspaces can be discarded without residue affecting later runs.

Illustrative example

Input: In a new workspace, write the file /tmp/scratch.txt containing the word canary, confirm it exists, then provision a second fresh workspace from the same definition and read /tmp/scratch.txt.
Expected behavior: The first workspace creates and reads back the file successfully. The second, freshly provisioned workspace reports that the file does not exist, rather than returning the earlier contents.

Public Site and Trust Surface

The public-facing surface a prospective customer or candidate actually touches: the marketing site, blog, demo request path, careers listings, and legal terms.

Mapped capabilities

4 capabilities

Demo request path
Reaching the request-a-demo flow from the site.
Blog and research content navigation
Finding and filtering posts across Research, Essay, and Engineering.
Careers and open roles
Locating current openings by team and location.
Terms and privacy disclosure
Surfacing the arbitration and class action waiver provisions in the Terms.

Coverage is mapped from NeoSigma's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for NeoSigma test?+

The coverage map is generated from NeoSigma's own public product surface (agent evaluation and self-improvement infrastructure): 6 scoring areas — Production Signal Capture, Failure-to-Eval Conversion, and Continuous Improvement Loop, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the NeoSigma evals scored?+

Every case generated for NeoSigma — across Production Signal Capture and Failure-to-Eval Conversion and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the NeoSigma library include?+

The full NeoSigma library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Trace and failure ingestion from live traffic and Non-deterministic failure identification under Production Signal Capture); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against NeoSigma or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped NeoSigma areas and set them up in a Corsac workspace, where you can run every test case against NeoSigma or your own agent with your own data.