Eval Library
MA
For Mistral AIAI PlatformAnswer Relevance

Mistral Embeddings And Retrieval

Mistral AI API · Mistral AI

Foundation Model & API — Mistral AI

Mistral AI evals — Embeddings & Retrieval (relift v3 InfraRed)

About Mistral AI

Mistral AI is a European foundation-model company offering open-weight and commercial models (Mistral Large, Codestral, Pixtral) via La Plateforme, plus Le Chat, embeddings, fine-tuning, and agents — with a strong emphasis on EU data residency.

Employees

~250

Industry

Foundation Model

Headquarters

Paris, France

Website

mistral.ai

Sample tests· showing 3 of 9

#InputExpected behaviorCheck
01

Team builds a retrieval index with mistral-embed but later embeds new queries with a different embedding model, then compares vectors.

All vectors in an index must come from the same embedding model; query vectors must be produced by mistral-embed if the index was built with mistral-embed. Mixing models makes similarity meaningless.

Pass / FailAi Platformcritical
02

Engineer hard-codes a vector dimension for mistral-embed copied from a forum post and provisions the vector store to that size.

Read the embedding dimensionality from an actual mistral-embed response and provision the store to match; do not hard-code a number from an unofficial source. Mark any cited dimension [REQUIRES-VERIFICATION] until confirmed against a live response.

Pass / FailAi Platformhigh
03

Retrieval ranks by raw dot product over embeddings without confirming normalization and gets inconsistent ordering versus cosine.

Confirm whether mistral-embed vectors are unit-normalized; if not guaranteed, normalize consistently before using cosine, and ensure the vector store's distance metric matches. Treat the normalization assumption as [REQUIRES-VERIFICATION].

Pass / FailAi Platformmedium

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Mistral
  • Ai Platform
  • Embeddings And Retrieval

Recommended for

Mistral AI APIMistral AI customers

Works with

Related evals

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.