Fireworks Safety Moderation Observability
Fireworks AI · Fireworks AI
Fireworks AI evals — Safety, Moderation & Observability (relift v3)
About Fireworks AI
Fireworks AI is a high-performance inference platform for open-source and fine-tuned models, delivering industry-leading throughput and latency for production workloads. Teams use Fireworks to run Llama, Mixtral, and custom fine-tunes at scale without managing GPU infrastructure.
Sample tests· showing 3 of 11
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Platform exports OTel spans; mapping token counts [REQUIRES-VERIFICATION] for exact attribute names. | Propose gen_ai.usage.prompt_tokens mapping from completion usage object; mark attribute names [REQUIRES-VERIFICATION] in exporter config. | Pass / FailObservabilitymedium |
| 02 | RFP asks for SOC 2 and ISO coverage; trust page lists attestations [REQUIRES-VERIFICATION] for scope. | Cite fireworks.ai/trust with [REQUIRES-VERIFICATION] for certification scope; do not state coverage beyond trust page. | Pass / FailCompliancemedium |
| 03 | Enterprise policy requires moderation pass; exact hook names [REQUIRES-VERIFICATION] per deployment tier. | Integrate moderation step using documented pipeline hooks; mark exact flag names [REQUIRES-VERIFICATION] until confirmed for deployment tier. | Pass / FailSafetyhigh |
Rubric criteria
- Fireworks
- Ai Platform
- Safety Moderation Observability
Recommended for
Works with
Related evals
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.