Eval Library
G
For GroqAI Platform

Structured Outputs And Json Mode

GroqCloud API · Groq

Fast Inference — Groq (GroqCloud)

Groq evals — Structured Outputs & JSON Mode (relift v3 InfraRed)

About Groq

Groq builds the LPU (Language Processing Unit) inference engine and GroqCloud — an OpenAI-compatible API that serves leading open models (Llama, Mixtral, Gemma, Qwen) at very high tokens-per-second with low, deterministic latency. Developers use GroqCloud for real-time chat, tool use, structured outputs, and speech-to-text without managing GPU infrastructure.

Employees

~300

Industry

AI Inference Platform

Headquarters

Mountain View, CA

Website

groq.com

Sample tests· showing 3 of 9

#InputExpected behaviorCheck
01

Agent sets response_format={type:'json_object'} but the prompt never instructs the model to produce JSON, and the model returns prose.

When using response_format json_object, also instruct the model in the prompt to respond with JSON (the documented requirement). json_object guarantees syntactically valid JSON but not a particular shape — pair it with a clear schema description in the prompt.

Pass / FailAi Platformhigh
02

A schema constrains status to enum ['open','closed'] but the model returns 'in_progress'.

Enforce enum membership on validation; an out-of-enum value is a failed result that should be rejected and retried (or repaired), not mapped to a default silently. Surface the violation so the prompt or schema can be tightened.

Pass / FailAi Platformmedium
03

Agent needs guaranteed schema-valid output and sets response_format with a json_schema, but does not check whether the chosen model supports structured outputs.

Use response_format={type:'json_schema', json_schema:{...}} on a model that supports structured outputs, and confirm support against current docs — not every GroqCloud model honors strict json_schema. Validate the returned object against the schema client-side as a backstop. [REQUIRES-VERIFICATION]…

Pass / FailAi Platformhigh

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Groq
  • Ai Platform
  • Structured Outputs And Json Mode

Recommended for

GroqCloud APIGroq customers

Works with

Related evals

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.