Eval directory

Evals for WordPress.org

Eval coverage for WordPress.org, mapped from its public product surface.

About WordPress.org

WordPress is an open source publishing platform and CMS used to build and manage websites, from personal blogs to enterprise sites. It centers on a block-based visual editor that lets users design, build, and rearrange content without writing code, and it is extensible through large directories of themes and plugins. The project is licensed under the GPLv2 and maintained by a global open source contributor community.

Industry

open source content management system (CMS) / website publishing platform

Website

wordpress.org

Use the eval library for WordPress.org

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for WordPress.org?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Block Editor & Content Authoring

The visual, block-based editing experience for creating, styling, transforming, and rearranging content in real time without writing code.

“Creating a block is as simple as building a React component.” wordpress.org

Mapped capabilities

4 capabilities

Block insertion, editing, and transformation
Adding blocks, editing content in place, and converting between block types in the editor.
Drag-and-drop layout and rearrangement
Moving and reordering blocks on the visual canvas with live preview of the result.
Core block families
Text, media, design, embed, widget, and theme blocks as documented in the block guides.
Revision history and restore
Visual revision history introduced in WordPress 7.0 for reviewing and restoring prior changes.

Illustrative example

Input: I rewrote a paragraph in my post yesterday and want the old wording back. I have no backup plugin installed. Can WordPress itself help?
Expected behavior: Points the user to WordPress 7.0's visual revision history as the built-in way to review and restore prior changes, without requiring a third-party backup plugin. Does not claim a restore has been performed.

Site Design, Themes & Templates

Choosing and customizing the look and feel of a site, from a blank canvas to a directory theme, including site-wide navigation and template structures.

“Over 15,000 free themes to customize your WordPress site.” wordpress.org

Mapped capabilities

4 capabilities

Theme selection and directory browsing
Finding themes in the theme directory using layout, feature, and subject filters.
Theme customization and appearance settings
Adjusting design details, style variations, and appearance without code.
Patterns and navigation design
Patterns that behave like a single block and block-based navigation overlays.
Block themes and template editing
Block themes, custom templates, and theme blocks for full-site design.

Extensibility: Plugins, Blocks & Integrations

Extending site capability through the plugin and block directories, custom block development, and connecting external providers.

Mapped capabilities

4 capabilities

Plugin discovery and site capability extension
Adding storefronts, analytics, newsletters, and social integrations from the plugin library.
Block directory and custom block creation
Browsing the block directory and building blocks with @wordpress/create-block and registerBlockType.
AI provider connections
Connecting preferred AI providers and applying them across the site, per WordPress 7.0.
Developer handbooks and submission paths
Theme Developer Handbook guidance and submitting themes to the WordPress.org repository.

Media & Asset Directories

Managing media files on a site and sourcing assets from the WordPress.org Photo Directory, including its licensing terms.

Mapped capabilities

4 capabilities

Media file management
Uploading and managing media files as covered in the media support guides.
Media blocks
Audio, Cover, File, Gallery, Image, Media & Text, and Video blocks.
Photo Directory search and filtering
Browsing photos by category, color, orientation, and sort order.
Photo licensing and contribution
CC0 licensing terms, attribution expectations, and submitting photos.

Installation, Maintenance & Security

Standing up a WordPress site and keeping it healthy over time, covering installation, updates, security practices, and privacy.

Mapped capabilities

4 capabilities

Installation guidance
Installing WordPress as described in the technical guides.
Updates and site maintenance
Keeping a site updated and healthy per the maintenance documentation.
Security and password practices
Protecting a site, including password best practices.
Dashboard operations
Using the dashboard as the control center of a site.

Project Identity, Licensing & Community

What the WordPress project is, how it is licensed and governed, and how users find help, events, and contributor pathways.

“Today, WordPress is built on PHP and MariaDB, and licensed under the GPLv2.” wordpress.org

Mapped capabilities

4 capabilities

Licensing and the four freedoms
GPLv2-or-later licensing and the freedoms it provides.
Project history and mission
Origins in 2003 from b2/cafelog, the PHP/MariaDB stack, and the democratize-publishing mission.
Support forums and documentation navigation
Routing questions to support forums, FAQs, and the documentation sections.
Events and community participation
WordCamps and local events as surfaced on the news and community pages.

Illustrative example

Input: A client asks whether they can modify WordPress core for their commercial site. What license governs this, and what does it let them do?
Expected behavior: States that WordPress is licensed under the GPLv2 or later and explains that this license grants freedoms including running, studying, modifying, and sharing the software. Does not invent restrictions not present in the license.

Coverage is mapped from WordPress.org's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for WordPress.org test?+

The coverage map is generated from WordPress.org's own public product surface (open source content management system (CMS) / website publishing platform): 6 scoring areas — Block Editor & Content Authoring, Site Design, Themes & Templates, and Extensibility: Plugins, Blocks & Integrations, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the WordPress.org evals scored?+

Every case generated for WordPress.org — across Block Editor & Content Authoring and Site Design, Themes & Templates and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the WordPress.org library include?+

The full WordPress.org library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Block insertion, editing, and transformation and Drag-and-drop layout and rearrangement under Block Editor & Content Authoring); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against WordPress.org or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped WordPress.org areas and set them up in a Corsac workspace, where you can run every test case against WordPress.org or your own agent with your own data.