Eval directory

Evals for Moderne

Eval coverage for Moderne, mapped from its public product surface.

About Moderne

Moderne parses every repository in a codebase into a Lossless Semantic Tree — a compiler-accurate model the site calls a code "genome" — so that types and relationships are fully resolved rather than inferred from raw text. On top of that model, agents distill fixes into deterministic recipes that apply the same change across one repo or many at once, with engineers validating the result. The site states Moderne was named a Leader in the first Gartner Magic Quadrant for AI-Augmented Code Modernization Tools.

Industry

AI-augmented code modernization and automated refactoring platform

Headquarters

Miami, FL, US

Website

moderne.ai

Use the eval library for Moderne

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Moderne?

6 scoring areas · 20 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Code genome & Lossless Semantic Tree

How the product explains and uses its compiler-accurate model of a codebase, where types and relationships are resolved rather than inferred from raw text.

“We sequence every repo into a Lossless Semantic Tree — a compiler-accurate model that delivers certainty, not probability.” moderne.ai

Mapped capabilities

4 capabilities

Sequencing repositories into an LST
Every repo in a codebase is parsed into the semantic model that the site calls a genome.
Resolved types and relationships
Answers distinguish resolved semantic facts from text-level pattern matching.
Certainty vs. probability framing
The model is presented as deterministic ground truth, not a probabilistic guess.
Model coverage across a codebase
Scope claims stay tied to repositories actually sequenced.

Deterministic recipes

Agents distill a fix into a recipe that applies the same change reproducibly, rather than re-deriving an edit per repository.

“Moderne gives humans and agents deterministic tools to drive code change accurately and at scale” moderne.ai

Mapped capabilities

3 capabilities

Distilling a fix into a recipe
An observed defect and its fix become a reusable, deterministic transformation.
Repeatable application
The same recipe produces the same edit given the same code.
Recipe vs. freeform agent edit
Distinguishing deterministic recipe application from ad hoc generated patches.

Illustrative example

Input: We found one bad date-parsing pattern in a service. How does Moderne get that same fix into the other 400 services that have it?
Expected behavior: Explains that the fix is distilled into a deterministic recipe that applies the identical change across every affected repo in one run, and that engineers validate the result before it is considered done.

Fleet-scale change

Applying a single change across one repository or many at once, which the site frames as ranging up to 100,000 repos.

“whether across a single repo or 100,000 of them” moderne.ai

Mapped capabilities

3 capabilities

Single-repo application
The same mechanism works for one repository.
Cross-repo fan-out
One recipe applied across many services in a single run.
Finding every affected site
Locating each occurrence of a defect across the sequenced codebase.

Agent + engineer workflow

The division of labor the site states explicitly: run by agents, validated by your engineers.

“Run by agents, validated by your engineers.” moderne.ai

Mapped capabilities

3 capabilities

Agent-initiated change
Agents read the genome and drive the change.
Engineer validation step
Results are reviewed and validated by engineers before they are treated as done.
Human-readable result review
Changes are presented so an engineer can inspect what was edited and why.

Claim & positioning integrity

Whether public-facing statements about Moderne match what the supplied evidence actually says, including the analyst recognition claim.

“Moderne named a Leader in the first Gartner® Magic Quadrant™ for AI-Augmented Code Modernization Tools.” moderne.ai

Mapped capabilities

3 capabilities

Gartner Leader claim wording
Named a Leader in the first Gartner Magic Quadrant for AI-Augmented Code Modernization Tools — stated as-is, not upgraded.
Company facts
Founders and Miami, FL location as published in the Organization schema.
No capability inflation
Claims stay inside what the evidence supports.

Web surface & machine readability

How Moderne presents itself to crawlers, social platforms, and answer engines through server-rendered metadata and host-gated indexing.

Mapped capabilities

4 capabilities

Social card metadata
og: and twitter: title, description, and the sitewide brand/og-default.png card.
Structured data graph
Organization and WebSite emitted on every page, with publisher resolved by @id.
Staging vs. production indexing
noindex,nofollow injected only on non-production hostnames.
Consent-gated analytics
Consent Mode v2 defaults denied until the CookiePro banner updates it; localhost skipped.

Illustrative example

Input: What image and title does moderne.ai use for its link preview card?
Expected behavior: Returns the sitewide branded card at https://www.moderne.ai/brand/og-default.png with the title "Moderne: your codebase has a genome", and notes the card type is summary_large_image.

Coverage is mapped from Moderne's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Moderne test?+

The coverage map is generated from Moderne's own public product surface (AI-augmented code modernization and automated refactoring platform): 6 scoring areas — Code genome & Lossless Semantic Tree, Deterministic recipes, and Fleet-scale change, and more — spanning 20 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Moderne evals scored?+

Every case generated for Moderne — across Code genome & Lossless Semantic Tree and Deterministic recipes and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Moderne library include?+

The full Moderne library is built on request. The coverage map spans 6 areas and 20 capabilities (for example, Sequencing repositories into an LST and Resolved types and relationships under Code genome & Lossless Semantic Tree); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Moderne or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Moderne areas and set them up in a Corsac workspace, where you can run every test case against Moderne or your own agent with your own data.