Mistral Json Mode And Structured Output
Mistral AI API · Mistral AI
Foundation Model & API — Mistral AI
Evaluates Mistral AI's JSON Mode & Structured Output across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
About Mistral AI
Mistral AI is a European foundation-model company offering open-weight and commercial models (Mistral Large, Codestral, Pixtral) via La Plateforme, plus Le Chat, embeddings, fine-tuning, and agents — with a strong emphasis on EU data residency.
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Operator sets response_format={type:'json_object'} and expects field-level schema guarantees for a downstream DB write. | json_object guarantees syntactically valid JSON, not your schema; use response_format json_schema for field guarantees, or post-validate every field. Choose based on whether structure must be enforced. | Pass / FailAi Platformhigh |
| 02 | Operator enables json_object but the system/user prompt never asks for JSON, and outputs are sometimes empty or whitespace. | When using json_object, also instruct the model in the prompt to produce JSON; the format flag constrains syntax but the prompt must still request the structure. Validate non-empty output. | Pass / FailAi Platformmedium |
| 03 | Agent needs guaranteed-parseable, schema-conformant JSON and uses response_format={type:'json_schema', json_schema:{...}} with the schema marked strict. | Supply a complete JSON Schema with required fields and additionalProperties:false; still wrap the parse in error handling. Verify the returned object conforms before the DB write. | Pass / FailAi Platformhigh |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Mistral
- Ai Platform
- Json Mode And Structured Output
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Mistral Json Mode And Structured Output eval for Mistral AI Mistral AI API test?+
Evaluates Mistral AI's JSON Mode & Structured Output across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
How is the Mistral Json Mode And Structured Output eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Mistral Json Mode And Structured Output pack for Mistral AI Mistral AI API contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Mistral Json Mode And Structured Output pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.