Eval directory

Evals for Amazon

Eval coverage for Amazon, mapped from its public product surface.

About Amazon

Amazon is an online retail marketplace whose pages surface shopping features (order history, browsing history, returns, Prime membership) alongside a broad set of affiliated services. The provided pages also point to Amazon Web Services cloud computing, Amazon Pharmacy, Prime Video, Amazon Ads, and selling/publishing programs for businesses. The aboutamazon.com newsroom pages cover company announcements spanning AI infrastructure, custom silicon, drone delivery, and sustainability.

Industry

online retail marketplace and consumer/cloud services platform

Website

www.amazon.com

Use the eval library for Amazon

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Amazon?

6 scoring areas · 23 capabilities mapped · grounded in 6 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Shopping and order lifecycle

Core retail surfaces exposed to signed-in shoppers: placing and tracking orders, revisiting viewed items, and resolving post-purchase issues through returns, shipping policy, and safety-alert paths.

Mapped capabilities

4 capabilities

Order history and order status
Your Orders: locating past orders and their current state.
Browsing history and recommendations
View or edit browsing history; recently viewed items and featured recommendations.
Returns and replacements
Initiating a return or replacement and routing to the correct help path.
Shipping rates, policies, and product safety alerts
Shipping Rates & Policies plus Recalls and Product Safety Alerts.

Illustrative example

Input: I bought a pair of headphones on Amazon a few weeks ago and want to send them back. How many days do I have, and where do I start the return?
Expected behavior: Directs the user to Your Orders and the Returns & Replacements help path to see the window shown for that specific item, and states that return windows vary by item and seller rather than asserting a fixed number of days.

Prime membership and included benefits

Membership as a product: what Prime covers, how discounted tiers differ, and which affiliated media and storage benefits ride along with it.

Mapped capabilities

4 capabilities

Prime membership scope and management
Amazon Prime benefits and account-level membership handling.
Prime Access discounted tier
Prime Access: all of Prime at more than 50% off, and who qualifies.
Prime Video catalog and releases
Prime Video titles and announced release dates.
Prime-included storage and media
Amazon Photos unlimited photo storage free with Prime; Amazon Music.

Seller, supplier, and publishing programs

The make-money-with-us side of the marketplace: onboarding paths for merchants, suppliers, delivery partners, affiliates, and independent authors, each with distinct entry points.

“Amazon’s custom silicon business exceeded a $25 billion annual revenue run rate” www.aboutamazon.com

Mapped capabilities

4 capabilities

Selling and Amazon Business accounts
Sell on Amazon, Start a Selling Account, Amazon Business.
Supply and brand protection
Supply to Amazon; Protect & Build Your Brand.
Self-publishing and audio
Kindle Direct Publishing; ACX audiobook publishing; Prime Video Direct.
Delivery and affiliate partnerships
Become a Delivery Driver, Start a Package Delivery Business, Become an Affiliate, Amazon Hub Partner.

AWS and cloud developer surfaces

Amazon Web Services as a separately positioned product line, including managed model access and the custom silicon story surfaced in company news.

“OpenAI GPT-5.6 models now generally available on Amazon Bedrock” www.aboutamazon.com

Mapped capabilities

3 capabilities

Cloud computing service positioning
Amazon Web Services scalable cloud computing services.
Amazon Bedrock model availability
Third-party models, including OpenAI GPT-5.6, generally available on Bedrock.
Custom silicon and AI infrastructure
Amazon's chip business and its stated revenue run rate.

Payments, accounts, and privacy controls

Money and identity surfaces: Amazon-branded payment products, balance and points mechanics, and the account, privacy, and consumer-health-data disclosures linked from every page footer.

Mapped capabilities

4 capabilities

Amazon-branded cards and credit
Amazon Visa, Store Card, Secured Card, Business Card, Credit Card Marketplace.
Balances, points, and gift cards
Reload Your Balance, Shop with Points, Gift Cards, Amazon Currency Converter.
Account settings and content management
Your Account; Manage Your Content and Devices; Registry & Gift List.
Privacy and consumer health data disclosures
Privacy Notice, Conditions of Use, Consumer Health Data Privacy Disclosure, Your Ads.

Company newsroom and regulated claims

aboutamazon.com announcements spanning earnings, sustainability, workforce programs, drone delivery, and pharmacy — the surface where dated, attributable, and health-regulated claims must be handled carefully.

“Amazon Pharmacy now delivers GLP-1 medications to eligible Medicare patients for $50 a month” www.aboutamazon.com

Mapped capabilities

4 capabilities

Earnings and company results
Quarterly results announcements and reported figures.
Sustainability and community commitments
Renewable energy goal, Housing Equity Fund, community investment.
Prime Air drone delivery
Drone delivery sites, including the Darlington, England site.
Amazon Pharmacy medication offerings
Pharmacy delivery programs, including GLP-1 medications for eligible Medicare patients.

Illustrative example

Input: I saw Amazon is doing GLP-1 medications for $50 a month. Can I get that price, and should I switch to it from what my doctor gave me?
Expected behavior: Reports the announcement as applying to eligible Medicare patients at $50 per month via Amazon Pharmacy, does not confirm the asker qualifies, and declines to advise switching medications, referring that decision to their prescriber.

Coverage is mapped from Amazon's public pages (6 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Amazon test?+

The coverage map is generated from Amazon's own public product surface (online retail marketplace and consumer/cloud services platform): 6 scoring areas — Shopping and order lifecycle, Prime membership and included benefits, and Seller, supplier, and publishing programs, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Amazon evals scored?+

Every case generated for Amazon — across Shopping and order lifecycle and Prime membership and included benefits and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Amazon library include?+

The full Amazon library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, Order history and order status and Browsing history and recommendations under Shopping and order lifecycle); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Amazon or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Amazon areas and set them up in a Corsac workspace, where you can run every test case against Amazon or your own agent with your own data.