Eval Library
MA
For Mistral AIAI Platform

Mistral Json Mode And Structured Output

Mistral AI API · Mistral AI

Foundation Model & API — Mistral AI

Mistral AI evals — JSON Mode & Structured Output (relift v3 InfraRed)

About Mistral AI

Mistral AI is a European foundation-model company offering open-weight and commercial models (Mistral Large, Codestral, Pixtral) via La Plateforme, plus Le Chat, embeddings, fine-tuning, and agents — with a strong emphasis on EU data residency.

Employees

~250

Industry

Foundation Model

Headquarters

Paris, France

Website

mistral.ai

Sample tests· showing 3 of 9

#InputExpected behaviorCheck
01

Operator sets response_format={type:'json_object'} and expects field-level schema guarantees for a downstream DB write.

json_object guarantees syntactically valid JSON, not your schema; use response_format json_schema for field guarantees, or post-validate every field. Choose based on whether structure must be enforced.

Pass / FailAi Platformhigh
02

Operator enables json_object but the system/user prompt never asks for JSON, and outputs are sometimes empty or whitespace.

When using json_object, also instruct the model in the prompt to produce JSON; the format flag constrains syntax but the prompt must still request the structure. Validate non-empty output.

Pass / FailAi Platformmedium
03

The mistralai SDK offers a typed parse path; the team hand-rolls json.loads and loses type validation.

Prefer the SDK's structured/typed parse where available so type mismatches surface as errors; hand-rolled json.loads silently accepts wrong-typed fields. Validate types at the boundary regardless.

Pass / FailAi Platformlow

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Mistral
  • Ai Platform
  • Json Mode And Structured Output

Recommended for

Mistral AI APIMistral AI customers

Works with

Related evals

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.