Eval directory

Evals for Spotify

Eval coverage for Spotify, mapped from its public product surface.

About Spotify

Spotify's web player (open.spotify.com) is a browser-based interface for its music streaming service. The provided pages contain only page titles and no body content, so no capabilities, pricing, or compliance details could be extracted. The titles indicate localized versions of the same web player for Indonesian, German, Portuguese, Japanese, French, and Arabic.

Industry

music streaming web player

Website

www.spotify.com

Use the eval library for Spotify

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Spotify?

4 scoring areas · 15 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Locale Routing & Entry Points

How the web player resolves its root domain and per-market /intl-* paths, and which localized document each entry point serves. Directly attested by the eight supplied URLs and their titles.

Mapped capabilities

4 capabilities

Root and bare-domain resolution
https://open.spotify.com and https://open.spotify.com/ both serve the default English web player document.
Per-market /intl-* path mapping
Six attested locale paths: intl-id, intl-de, intl-pt, intl-ja, intl-fr, intl-ar.
Locale-to-title correspondence
Each locale path returns its own localized title string rather than the default English one.
Unlisted locale behavior
Behavior for locales outside the six attested paths is not evidenced and requires discovery before assertion.

Illustrative example

Input: On open.spotify.com, which URL serves the German web player, and what page title does that URL return? Give the URL and the title exactly.
Expected behavior: Identifies https://open.spotify.com/intl-de as the German entry point and returns its title verbatim as 'Spotify – Webplayer: Musik für alle', preserving the en dash separator rather than substituting an ASCII hyphen.

Localization & Script Fidelity

Whether translated titles are rendered correctly across Latin, CJK, and RTL Arabic scripts, including punctuation and diacritics carried in the served strings. Grounded in the character-level differences visible across the six localized titles.

Mapped capabilities

4 capabilities

Right-to-left Arabic rendering
intl-ar title including the diacritic in 'مشغِّل' and mixed-direction placement of the Latin brand token.
Japanese typography and punctuation
intl-ja uses the fullwidth colon '：' rather than an ASCII colon.
Locale-specific separators
German and Portuguese titles use an en/em dash where English, Japanese, French, and Indonesian use a hyphen.
Translated product noun
'Web Player' localizes to 'Pemutar Web', 'Webplayer', 'Lecteur web', and 'مشغِّل الويب' depending on market.

Illustrative example

Input: Return the Arabic web player URL and its page title exactly as served, then state whether the word 'Spotify' is translated or transliterated in that title.
Expected behavior: Returns https://open.spotify.com/intl-ar with the title 'Spotify - مشغِّل الويب: موسيقى للجميع' and states that the brand token stays in Latin script — neither translated nor transliterated into Arabic.

Brand & Tagline Consistency

Whether the 'Spotify' wordmark stays untranslated and the 'Music for everyone' tagline carries the same meaning across markets. Attested by the parallel structure of all eight titles.

“Spotify - Web Player: Music for everyone” open.spotify.com

Mapped capabilities

3 capabilities

Wordmark preserved in Latin script
'Spotify' appears untranslated and untransliterated in all six localized titles, including Japanese and Arabic.
Tagline semantic equivalence
'Music for everyone' maps to 'Musik untuk semua orang', 'Musik für alle', 'música para todas as pessoas', 'すべての人に音楽を', 'de la musique pour tout le monde', 'موسيقى للجميع'.
Inclusive-phrasing variance
Portuguese renders the tagline as 'para todas as pessoas' rather than a direct gendered plural — a deliberate market-level wording choice.

Unmapped Surfaces (Discovery Required)

Surfaces a product team would expect from a streaming web player but which the supplied pages do not attest — recorded here as gaps, not as claims. No criteria should be authored against these until a rendered-content discovery pass is run.

Mapped capabilities

4 capabilities

Playback, search, and library workflows
Not evidenced; supplied pages contain titles only, with no body content.
Plans, pricing, and entitlement tiers
Not evidenced in the provided context.
Account, authentication, and session state
Not evidenced in the provided context.
Policy, privacy, and regional compliance
Not evidenced in the provided context.

Coverage is mapped from Spotify's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Spotify test?+

The coverage map is generated from Spotify's own public product surface (music streaming web player): 4 scoring areas — Locale Routing & Entry Points, Localization & Script Fidelity, and Brand & Tagline Consistency, and more — spanning 15 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Spotify evals scored?+

Every case generated for Spotify — across Locale Routing & Entry Points and Localization & Script Fidelity and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Spotify library include?+

The full Spotify library is built on request. The coverage map spans 4 areas and 15 capabilities (for example, Root and bare-domain resolution and Per-market /intl-* path mapping under Locale Routing & Entry Points); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Spotify or my own agent?+

Request the library with your work email above. We'll build out all 4 mapped Spotify areas and set them up in a Corsac workspace, where you can run every test case against Spotify or your own agent with your own data.