Structured Outputs And Json Schema
OpenAI API · OpenAI
Foundation Model & API — OpenAI (GPT)
Evaluates OpenAI's Structured Outputs & JSON Schema across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
About OpenAI
OpenAI builds the GPT model family and the OpenAI API — Responses and Chat Completions, function calling, Structured Outputs, embeddings, fine-tuning, the Batch API, moderation, the Realtime API, and the Agents SDK — used by developers to build AI products at scale.
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Agent needs guaranteed-parseable JSON for a downstream DB write and uses response_format json_schema with strict:true. | With strict:true the output conforms to the supplied JSON Schema; still wrap json.loads in error handling for refusal cases. Verify required/additionalProperties:false are set as strict mode requires. | Pass / FailAi Platformhigh |
| 02 | A structured-output request for disallowed content returns a populated refusal field instead of schema content. | Check the refusal field before parsing content; when present, surface the refusal and do not attempt json.loads on absent content. | Pass / FailAi Platformhigh |
| 03 | Schema uses patternProperties and minContains, which strict mode does not support. | Pre-validate the schema against strict-mode-supported keywords; on unsupported features, either restructure the schema or fall back to json_object with post-validation. Do not ship a schema the API will reject. | Pass / FailAi Platformhigh |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Openai
- Ai Platform
- Structured Outputs And Json Schema
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Structured Outputs And Json Schema eval for OpenAI OpenAI API test?+
Evaluates OpenAI's Structured Outputs & JSON Schema across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
How is the Structured Outputs And Json Schema eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Structured Outputs And Json Schema pack for OpenAI OpenAI API contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Structured Outputs And Json Schema pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.