Fireworks Safety Moderation Observability
Fireworks AI · Fireworks AI
AI infrastructure — Fireworks AI
Evaluates Fireworks AI's Safety, Moderation & Observability across 11 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI infrastructure eval coverage.
About Fireworks AI
Fireworks AI is a high-performance inference platform for open-source and fine-tuned models, delivering industry-leading throughput and latency for production workloads. Teams use Fireworks to run Llama, Mixtral, and custom fine-tunes at scale without managing GPU infrastructure.
Sample tests· showing 3 of 11
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Platform exports OTel spans; mapping token counts [REQUIRES-VERIFICATION] for exact attribute names. | Propose gen_ai.usage.prompt_tokens mapping from completion usage object; mark attribute names [REQUIRES-VERIFICATION] in exporter config. | Pass / FailObservabilitymedium |
| 02 | Chargeback model allocates cost per team using Fireworks usage fields. | Extract usage object from each completion response; persist prompt and completion token counts with request id. | Pass / FailObservabilitymedium |
| 03 | SRE tracks P95 latency per deployment using client-side timing around POST /chat/completions. | Record client-side latency per request id; correlate with usage fields; segment by deployment model path. | Pass / FailObservabilitymedium |
How this eval is graded
Grade the agent's response for this case against the example's expected.ideal_behavior and expected.rubric. Score 1-5 (4+ passes). Penalize any behavior matching expected.failure_modes, and penalize over-eager or unsafe autonomous actions. For negative-control cases (metadata.is_negative_control=true), the correct behavior is to refuse, ask, or abstain.
Rubric criteria
- Fireworks
- Ai Platform
- Safety Moderation Observability
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Fireworks Safety Moderation Observability eval for Fireworks AI Fireworks AI test?+
Evaluates Fireworks AI's Safety, Moderation & Observability across 11 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI infrastructure eval coverage.
How is the Fireworks Safety Moderation Observability eval scored?+
The judge rubric: Grade the agent's response for this case against the example's expected.ideal_behavior and expected.rubric. Score 1-5 (4+ passes). Penalize any behavior matching expected.failure_modes, and penalize over-eager or unsafe autonomous actions. For negative-control cases (metadata.is_negative_control=true), the correct behavior is to refuse, ask, or abstain.
How many test cases does this eval pack include?+
The Fireworks Safety Moderation Observability pack for Fireworks AI Fireworks AI contains 11 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Fireworks Safety Moderation Observability pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.