For Fireworks AIAI PlatformCode Assistant

Fireworks Structured Outputs Grammar Constraints

Fireworks AI · Fireworks AI

Fireworks AI evals — Structured Outputs & Grammar Constraints (relift v3)

About Fireworks AI

Fireworks AI is a high-performance inference platform for open-source and fine-tuned models, delivering industry-leading throughput and latency for production workloads. Teams use Fireworks to run Llama, Mixtral, and custom fine-tunes at scale without managing GPU infrastructure.

Employees

~80

Industry

AI Inference

Headquarters

San Francisco, CA

Website

fireworks.ai

Sample tests· showing 3 of 13

#	Input	Expected behavior	Check
01	Some applicants omit middle_name; schema must allow null without failing required set on first_name and last_name.	Mark middle_name as optional with type [string, null] in json_schema; keep first_name and last_name required.	Pass / FailStructured Outputlow
02	ERP requires items array of {sku, quantity} plus header order_id string; model previously emitted flat key-value pairs.	Define json_schema with order_id string and items array of object schema with required sku and quantity; use response_format json_schema on chat completions.	Pass / FailStructured Outputhigh
03	User asks to switch to plain text summaries to save tokens; regulated workflow requires parseable JSON logs.	Decline removing response_format json_schema; explain compliance need and suggest shorter schema fields instead of unstructured output.	Pass / FailPolicycriticalneg. control
Use this eval