Eval directory

Evals for Automattic

Eval coverage for Automattic, mapped from its public product surface.

About Automattic

Automattic is the company behind WordPress.com, WooCommerce, Tumblr, Jetpack, Simplenote, Day One, Pocket Casts, Akismet, Newspack, Beeper, and WordPress VIP. Its stated mission is to democratize publishing, commerce, and messaging through open-web tools, many offered free or as open source. It sells premium hosting, plugins, and enterprise services on top of the open source WordPress project it contributes to.

Employees

more than 1,447

Industry

web publishing, commerce, and messaging platform company (WordPress.com, WooCommerce, Tumblr, Jetpack)

Headquarters

San Francisco, CA, USA

Website

automattic.com

Use the eval library for Automattic

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Automattic?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Publishing platforms

Guidance across Automattic's publishing surfaces — WordPress.com hosting and plans, Tumblr, Longreads, The Atavist, Newspack for news publishers, and the 100 Year Plan — including which one fits a given creator's goal.

“Premium WordPress.com hosting with a century-long commitment, designed to preserve digital legacy” automattic.com

Mapped capabilities

4 capabilities

WordPress.com site creation and plan tiers
Free hosted blog/site onboarding, premium upgrades, domains, and the 100 Year Plan's stated century-long hosting and domain renewal commitment.
Newspack for independent publishers
Positioning for newsrooms: content management, audience building, and reader-revenue/donation tooling as described in published customer stories.
Tumblr and social-style publishing
Interest-based posting and community surfaces, and when Tumblr rather than WordPress.com is the right recommendation.
Editing and authoring tools
Gutenberg/block editing, WordPress Studio, mobile apps for iOS and Android, and the AI Website Builder as named products.

Commerce

WooCommerce and the surrounding commerce stack for merchants selling online, including the free-plugin baseline and what is layered on top.

Mapped capabilities

4 capabilities

WooCommerce store setup
Installing the free plugin on a WordPress site, product pages, and the boundary between core plugin and paid extensions.
Merchant growth and marketing
MailPoet email, Blaze/WordAds promotion, and Crowdsignal feedback as commerce-adjacent products.
Course and membership selling
Sensei for course-based products and how it relates to a WooCommerce storefront.
Release and feature currency
Answering about WooCommerce releases only from published release notes, without inventing version numbers or ship dates.

Messaging and personal apps

Automattic's consumer and messaging products — Beeper, Simplenote, Day One, Pocket Casts, Gravatar — which sit outside the WordPress stack and are frequently confused with it.

Mapped capabilities

4 capabilities

Beeper unified messaging
Consolidating fragmented chat networks into one client, per Automattic's stated messaging mission.
Journaling and notes
Day One for private journaling and Simplenote for lightweight notes, including their distinct privacy framing.
Pocket Casts podcast listening
Podcast playback and show discovery for listeners.
Gravatar profile identity
One globally reused avatar/profile image across sites and services.

Site protection and integrity

The trust, safety, and reliability products a site owner relies on: spam filtering, security scanning, backups, and performance.

“hosted version of the open source WordPress software with added features for security, speed and support” automattic.com

Mapped capabilities

4 capabilities

Akismet spam filtering
Comment and form spam detection as the correct product to route a spam complaint to.
Jetpack security, growth, and speed
The bundled plugin's stated safety, growth, and speed roles on a self-hosted WordPress site.
Backups and vulnerability scanning
VaultPress backups and WPScan/WP Super Cache as named protection and performance tooling.
Incident routing and support paths
Directing a site owner to the right product or support channel rather than improvising a fix or an SLA.

Illustrative example

Input: My WordPress site is drowning in junk comments every morning. Which Automattic product handles that?
Expected behavior: The assistant names Akismet as the spam filtering product and briefly explains it filters comment spam. It may mention Jetpack as the broader security, growth, and speed plugin, but does not present an unrelated product as the answer.

Enterprise, hosting infrastructure, and agencies

The paid enterprise and partner tier built on top of open-source WordPress: WordPress VIP, Pressable, WP Cloud, Automattic for Agencies, and Parse.ly analytics.

Mapped capabilities

4 capabilities

WordPress VIP enterprise positioning
Enterprise WordPress for large organizations, with named public references such as Meta, Salesforce, and Slack used only as published.
Hosting and platform infrastructure
Pressable and WP Cloud as hosting surfaces distinct from WordPress.com's consumer plans.
Agency partner program
Automattic for Agencies as the channel for partners managing client sites.
Analytics for publishers
Parse.ly as the content analytics product within the portfolio.

Company facts and open-source boundaries

Accuracy discipline unique to Automattic: separating the WordPress open-source project from Automattic's commercial products, honoring published naming guidance, and citing company figures only as stated.

“A fully distributed company, Automattic has more than 1,447 employees working from 81 countries.” automattic.com

Mapped capabilities

4 capabilities

WordPress.org vs. WordPress.com disambiguation
Distinguishing the open-source project maintained by independent contributors from Automattic's hosted service, including the published request to append '.com' for the product.
Open-source contribution framing
Automattic as a major contributor to — not the owner of — WordPress core, plus Five for the Future, BuddyPress, bbPress, P2, and WordCamps.
Corporate and press facts
Founding in August 2005, founder and CEO Matt Mullenweg, San Francisco headquarters, privately held status, and staff/country counts stated only as published on the press page.
Distributed-work and mission claims
The fully distributed model and the democratize publishing, commerce, and messaging mission, without embellishing beyond published wording.

Illustrative example

Input: I want to write about how Automattic owns WordPress and controls the whole thing. Can you confirm that for my article?
Expected behavior: The assistant corrects the premise: WordPress is open source software maintained by thousands of independent contributors, and Automattic is a major contributor that separately owns and operates the hosted WordPress.com service. It uses the '.com' suffix when naming the product.

Coverage is mapped from Automattic's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Automattic test?+

The coverage map is generated from Automattic's own public product surface (web publishing, commerce, and messaging platform company (WordPress.com, WooCommerce, Tumblr, Jetpack)): 6 scoring areas — Publishing platforms, Commerce, and Messaging and personal apps, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Automattic evals scored?+

Every case generated for Automattic — across Publishing platforms and Commerce and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Automattic library include?+

The full Automattic library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, WordPress.com site creation and plan tiers and Newspack for independent publishers under Publishing platforms); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Automattic or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Automattic areas and set them up in a Corsac workspace, where you can run every test case against Automattic or your own agent with your own data.