
Json Structured Output
DeepSeek API · DeepSeek
Foundation Model & API — DeepSeek
Evaluates DeepSeek's JSON / Structured Output across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
About DeepSeek
DeepSeek is an AI company shipping frontier open-weight models (DeepSeek-V3, DeepSeek-R1) and an OpenAI-compatible API with a separate reasoner model (deepseek-reasoner), automatic disk-based context caching, function calling, JSON output, and very low token pricing. The models are released under an MIT license alongside the hosted API.
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | An integrator wants strict JSON back and sets response_format to an invented {type:'json_schema'} value DeepSeek does not document, then treats the error as a bug. | Set response_format={type:'json_object'} — the documented DeepSeek JSON-output mode. The model then returns a JSON-parseable string. Do not pass undocumented response_format variants. | Pass / FailAi Platformhigh |
| 02 | response_format=json_object is set but the prompt never mentions JSON or shows the desired shape; the output is empty or non-JSON. | Per docs, when using json_object the prompt must include the word 'json' and describe the expected schema/keys. Add an explicit instruction and an example shape; do not rely on response_format alone to induce well-formed JSON. | Pass / FailAi Platformhigh |
| 03 | The JSON-mode response is fed straight into json.loads with no error handling; an occasional malformed payload crashes the worker. | Parse JSON-mode output defensively — wrap json.loads, validate against the expected schema, and on failure retry or route to a fallback. json_object makes valid JSON likely but the client must still guard against parse and schema-mismatch errors. | Pass / FailAi Platformhigh |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Deepseek
- Ai Platform
- Json Structured Output
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Json Structured Output eval for DeepSeek DeepSeek API test?+
Evaluates DeepSeek's JSON / Structured Output across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
How is the Json Structured Output eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Json Structured Output pack for DeepSeek DeepSeek API contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Json Structured Output pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.