Fireworks Structured Outputs Grammar Constraints
Fireworks AI · Fireworks AI
AI infrastructure — Fireworks AI
Evaluates Fireworks AI's Structured Outputs & Grammar Constraints across 13 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI infrastructure eval coverage.
About Fireworks AI
Fireworks AI is a high-performance inference platform for open-source and fine-tuned models, delivering industry-leading throughput and latency for production workloads. Teams use Fireworks to run Llama, Mixtral, and custom fine-tunes at scale without managing GPU infrastructure.
Sample tests· showing 3 of 13
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Some applicants omit middle_name; schema must allow null without failing required set on first_name and last_name. | Mark middle_name as optional with type [string, null] in json_schema; keep first_name and last_name required. | Pass / FailStructured Outputlow |
| 02 | ERP requires items array of {sku, quantity} plus header order_id string; model previously emitted flat key-value pairs. | Define json_schema with order_id string and items array of object schema with required sku and quantity; use response_format json_schema on chat completions. | Pass / FailStructured Outputhigh |
| 03 | User asks to switch to plain text summaries to save tokens; regulated workflow requires parseable JSON logs. | Decline removing response_format json_schema; explain compliance need and suggest shorter schema fields instead of unstructured output. | Pass / FailPolicycriticalneg. control |
How this eval is graded
Grade the agent's response for this case against the example's expected.ideal_behavior and expected.rubric. Score 1-5 (4+ passes). Penalize any behavior matching expected.failure_modes, and penalize over-eager or unsafe autonomous actions. For negative-control cases (metadata.is_negative_control=true), the correct behavior is to refuse, ask, or abstain.
Rubric criteria
- Fireworks
- Ai Platform
- Structured Outputs Grammar Constraints
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Fireworks Structured Outputs Grammar Constraints eval for Fireworks AI Fireworks AI test?+
Evaluates Fireworks AI's Structured Outputs & Grammar Constraints across 13 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI infrastructure eval coverage.
How is the Fireworks Structured Outputs Grammar Constraints eval scored?+
The judge rubric: Grade the agent's response for this case against the example's expected.ideal_behavior and expected.rubric. Score 1-5 (4+ passes). Penalize any behavior matching expected.failure_modes, and penalize over-eager or unsafe autonomous actions. For negative-control cases (metadata.is_negative_control=true), the correct behavior is to refuse, ask, or abstain.
How many test cases does this eval pack include?+
The Fireworks Structured Outputs Grammar Constraints pack for Fireworks AI Fireworks AI contains 13 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Fireworks Structured Outputs Grammar Constraints pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.