Eval directory

Evals for Meta

Eval coverage for Meta, mapped from its public product surface.

About Meta

Meta Store is Meta's online storefront for its consumer hardware, spanning AI glasses, Meta glasses, and Meta Quest headsets. The site is localized across multiple regions including Australia, Canada, Ireland, India, and Iceland. Its blog covers Meta Quest software, including game launches and the Meta Horizon+ subscription library.

Industry

consumer AI glasses and VR headset storefront

Website

www.meta.com

Use the eval library for Meta

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Meta?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Hardware Catalog & Product Discovery

Navigating and distinguishing the three consumer hardware lines the storefront sells — AI glasses, Meta glasses, and Meta Quest headsets — and answering shopper questions about which line a request maps to.

“Shop AI Glasses, Meta Glasses & Quest Headsets” www.meta.com

Mapped capabilities

4 capabilities

Product line disambiguation
Correctly routes a shopper's described need to AI glasses, Meta glasses, or Quest headsets without conflating the lines.
Catalog scope boundaries
Declines or flags requests for hardware or services outside the storefront's evidenced catalog.
Storefront entry navigation
Directs a visitor from the meta.com landing surface to the relevant product category.
Cross-line comparison framing
Compares the hardware lines on evidenced attributes only, without inventing specs, prices, or availability.

Regional Storefronts & Localization

The storefront is published across multiple localized regions — Australia, Canada, Ireland, India, and Iceland — each with its own path and locale. Covers correct region routing and locale-consistent presentation.

Mapped capabilities

4 capabilities

Region path routing
Maps a stated country to its correct storefront path (/au/, /ca/, /ie/, /in/, /is/) or the default surface.
Locale label consistency
Uses the locale-appropriate title casing and phrasing evidenced for each regional page.
Unsupported region handling
States that a region is not among the evidenced localized storefronts rather than fabricating a path.
Region-scoped claim hygiene
Avoids asserting region-specific pricing, stock, or shipping that the storefront context does not establish.

Illustrative example

Input: I'm in Ireland and want to browse Quest headsets on the Meta Store. What's the right site for me?
Expected behavior: Points the shopper to the Irish storefront at https://www.meta.com/ie/ as the localized Meta Store surface, without asserting Ireland-specific pricing, stock, or shipping terms that the storefront context does not establish.

Blog & Content Feed

The Meta.com Blog and its RSS feed publish dated posts covering Quest software, game launches, music content, and monthly recaps. Covers faithful retrieval, dating, and linking of feed items.

“Co-op action-adventure game Raider Legends launches today on Meta Quest” www.meta.com

Mapped capabilities

4 capabilities

Feed item retrieval
Returns the correct post for a described topic from the published feed entries.
Publication date accuracy
Reports post dates as published in the feed, including relative ordering of recent items.
Canonical link fidelity
Cites the post's own blog URL rather than a constructed or approximate link.
Recap vs. launch post distinction
Separates monthly roundup posts from individual title-launch announcements.

Quest Software & Title Announcements

Blog coverage of what is playable on Meta Quest — new game launches, in-title content drops, and monthly release recaps — and answering owner questions grounded in those announcements.

Mapped capabilities

4 capabilities

Launch announcement grounding
Describes a launching title using only the details stated in its announcement post.
In-title content updates
Handles free track, DLC, and update announcements distinctly from full game launches.
Monthly release recap lookup
Points to the correct month's recap post when asked what released in a given period.
Unannounced title restraint
Declines to confirm titles, dates, or platforms not present in the published posts.

Meta Horizon+ Subscription Library

The Meta Horizon+ subscription and its rotating game library, including monthly additions and the XBOX Game Pass starter-edition tie-in announced in the feed.

“Meta Horizon+ offers a huge library of great games you can play whenever you want” www.meta.com

Mapped capabilities

4 capabilities

Monthly library additions
Reports the number and framing of titles added in a given month as announced.
XBOX Game Pass inclusion
Characterizes the starter-edition Game Pass tie-in and its 50+ title library accurately.
Subscription vs. purchase distinction
Separates subscription-library access from individually purchased storefront titles.
Entitlement claim restraint
Avoids stating pricing, tiers, or regional availability the announcements do not specify.

Illustrative example

Input: What did Meta Horizon+ add this month, and is Xbox Game Pass part of it now?
Expected behavior: States that Horizon+ added five new games and that a starter edition of XBOX Game Pass brings a library of 50+ titles playable in-headset. Does not claim a full Game Pass tier, a price, or region-specific availability.

Grounding & Refusal Discipline

Cross-cutting behavior for a storefront-plus-blog surface: answering only from published product and post content, and saying so plainly when the answer is not covered.

Mapped capabilities

3 capabilities

Out-of-scope deferral
Redirects account, order-status, and support questions the published surfaces do not cover.
Fabrication resistance
Declines to invent specs, prices, release dates, or region availability under pressure.
Staleness acknowledgment
Notes that catalog and library contents change rather than presenting a snapshot as permanent.

Coverage is mapped from Meta's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Meta test?+

The coverage map is generated from Meta's own public product surface (consumer AI glasses and VR headset storefront): 6 scoring areas — Hardware Catalog & Product Discovery, Regional Storefronts & Localization, and Blog & Content Feed, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Meta evals scored?+

Every case generated for Meta — across Hardware Catalog & Product Discovery and Regional Storefronts & Localization and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Meta library include?+

The full Meta library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, Product line disambiguation and Catalog scope boundaries under Hardware Catalog & Product Discovery); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Meta or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Meta areas and set them up in a Corsac workspace, where you can run every test case against Meta or your own agent with your own data.