Eval directory

Evals for Microsoft

Eval coverage for Microsoft, mapped from its public product surface.

About Microsoft

Microsoft is a global technology company and a leading cloud and AI provider. Microsoft Copilot embeds AI assistance across Microsoft 365, Azure, and Teams — helping employees generate content, analyze data, and automate tasks across the Microsoft ecosystem.

Employees

~221,000

Industry

Enterprise Software & Cloud

Headquarters

Redmond, WA

Website

microsoft.com

Use the eval library for Microsoft

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Microsoft?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Microsoft 365 Plans, Pricing, and Billing

The plan comparison surface for individuals, families, students, business, schools, enterprise, frontline, nonprofit, and government — including what each tier bundles (apps, Copilot, cloud storage, security) and how monthly versus yearly billing changes the price.

“Microsoft 365 delivers cloud storage, advanced security, and Microsoft Copilot in your favorite apps—all in one plan.” www.microsoft.com

Mapped capabilities

4 capabilities

Segment-to-plan routing
Mapping a stated buyer profile (single user, family, student, small business, enterprise, nonprofit, government) to the plan family the site actually offers for that segment.
Billing cadence and stated savings
Explaining monthly versus yearly pricing and the site's stated annual savings without inventing discounts or extrapolating unlisted totals.
What's bundled per tier
Distinguishing the apps, Copilot access, storage, and security included at each tier, including Premium upgrade positioning for existing subscribers.
Office 365 to Microsoft 365 naming
Handling legacy 'Office 365' phrasing and directing users to the current Microsoft 365 equivalents without implying a product that no longer carries that name.

Illustrative example

Input: I'm a single user who wants Word, Excel, Copilot, and cloud storage. Which Microsoft 365 plan is the entry point, and does paying yearly instead of monthly save money?
Expected behavior: Identifies Microsoft 365 Personal at $9.99/month as the individual plan bundling apps, Copilot, storage, and security, and states that annual billing saves up to 17% versus monthly, using the site's stated figure rather than computing a new one.

Copilot as the Cross-Product AI Assistant

Copilot's positioning as an everyday AI companion embedded across apps, devices, and the phone — covering Copilot Chat, AI agents, prompt guidance, and the learning resources Microsoft points users to.

“Copilot helps you write, create, learn, shop, and stay organized with personalized AI support across your devices” www.microsoft.com

Mapped capabilities

4 capabilities

Copilot capability framing
Describing what Copilot is marketed to do — write, create, learn, shop, stay organized, find information — without asserting unlisted capabilities or model claims.
Where Copilot is available
Clarifying access across select apps, devices, and mobile, and which subscription tiers include Copilot versus require an upgrade.
Copilot Chat, agents, and prompt guidance
Routing questions about Copilot Chat, AI agents, the daily prompt guide, and the Copilot learning center to the correct resource.
Getting started and install path
Giving an accurate first step (try, download, or learn more) rather than a generic or fabricated onboarding flow.

Education: Teams and Microsoft 365 Education

The institution-facing surface where Teams for Education and Microsoft 365 Education are sold as secure, institution-governed platforms with AI-assisted learning tools, eligibility tied to a valid institutional email.

“it’s free with a valid, active institution email address” www.microsoft.com

Mapped capabilities

4 capabilities

Eligibility and free access
Stating that Teams for Education is free with a valid, active institution email address, and what that does and does not cover.
Institution governance and security framing
Explaining that AI-powered learning is kept securely governed by the institution, without overstating specific compliance or data claims.
Classroom feature surface
Covering assignments, class materials, notes, video collaboration and chat, plus Learning Accelerators, accessibility tools, and social-emotional learning resources.
Audience routing within Education
Directing K-12 leaders, higher-ed leaders, IT leaders, educators, and students to the resource path each has on the Education site.

Illustrative example

Input: Our school's IT lead asks whether teachers must purchase licenses before using Microsoft Teams for classes, assignments, and class materials.
Expected behavior: Answers that Teams for Education is free with a valid, active institution email address, so no purchase is required to get started, and points to downloading Teams with that email as the next step.

Product Discovery and Guided Navigation

The AI-assisted entry surface on microsoft.com — ask-a-question prompts, 'help me choose a PC', product-and-apps catalog, and the sitemap — where a visitor's vague intent must resolve to a real product page.

Mapped capabilities

4 capabilities

Intent-to-product resolution
Turning an underspecified need into the correct product or app in the catalog rather than a generic portfolio pitch.
Guided PC and device selection
Handling 'help me choose' style requests using only attributes the site presents, and asking a clarifying question when the request underdetermines the answer.
Portfolio disambiguation
Separating adjacent Microsoft properties (Microsoft 365 vs Office, Teams vs Skype, Windows 365 vs Windows 11, Edge vs Copilot) when a query is ambiguous.
Account and support handoff
Routing billing, account, admin self-help, and support-plan questions to the right support surface instead of answering as if authoritative.

Devices, Store Offers, and Commerce

The retail surface covering Surface, PCs, Xbox hardware and Game Pass, accessories, VR and mixed reality, certified refurbished, trade-in, and time-bound promotions such as back-to-school and bundle offers.

Mapped capabilities

4 capabilities

Promotion terms and conditions
Representing offers such as the student discount and the college PC bundle with their stated qualifiers, avoiding unstated eligibility or guaranteed value claims.
Hardware lineup positioning
Describing Surface and PC options using the site's own framing, without inventing specs, availability, or comparisons.
Xbox and Game Pass tiers
Distinguishing Xbox Game Pass Ultimate, PC Game Pass, console versus PC games, and accessories within the gaming catalog.
Trade-in and refurbished paths
Explaining certified refurbished and trade-in-for-cash as distinct commerce flows and where each is initiated.

Cloud, Security, and IT Administration

The business and institutional surface spanning Azure, Microsoft Security, Windows 365, Dynamics 365, Power Platform, digital sovereignty, the marketplace, and the admin and partner support paths that serve IT buyers.

“Microsoft 365 Education is a secure, integrated platform that supports learning today” www.microsoft.com

Mapped capabilities

4 capabilities

Business vs enterprise routing
Sending small business, enterprise, frontline, nonprofit, and government buyers to the correct plan family and sales path.
Security and sovereignty positioning
Conveying Microsoft Security and digital sovereignty offerings at the level of specificity the site supports, flagging anything beyond it as unverified.
Admin and partner support routing
Directing admin self-help, support plans, find-a-partner, and contact-sales requests to the correct channel.
Developer and learning resources
Pointing to Microsoft Learn, Developer, Visual Studio, Tech Community, and Marketplace for build-and-learn intent rather than consumer pages.

Coverage is mapped from Microsoft's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Microsoft test?+

The coverage map is generated from Microsoft's own public product surface (productivity, cloud, and AI software and devices): 6 scoring areas — Microsoft 365 Plans, Pricing, and Billing, Copilot as the Cross-Product AI Assistant, and Education: Teams and Microsoft 365 Education, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Microsoft evals scored?+

Every case generated for Microsoft — across Microsoft 365 Plans, Pricing, and Billing and Copilot as the Cross-Product AI Assistant and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Microsoft library include?+

The full Microsoft library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Segment-to-plan routing and Billing cadence and stated savings under Microsoft 365 Plans, Pricing, and Billing); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Microsoft or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Microsoft areas and set them up in a Corsac workspace, where you can run every test case against Microsoft or your own agent with your own data.