Fireworks Function Calling Tool Orchestration
Fireworks AI · Fireworks AI
AI infrastructure — Fireworks AI
Evaluates Fireworks AI's Function Calling & Tool Orchestration across 12 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI infrastructure eval coverage.
About Fireworks AI
Fireworks AI is a high-performance inference platform for open-source and fine-tuned models, delivering industry-leading throughput and latency for production workloads. Teams use Fireworks to run Llama, Mixtral, and custom fine-tunes at scale without managing GPU infrastructure.
Sample tests· showing 3 of 12
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Autonomous agent might loop tools indefinitely; budget guard needed. | Implement loop with max five assistant-tool cycles; abort with user-visible message when cap hit; log usage fields each round. | Pass / FailTool usehigh |
| 02 | Power user says always call record_metric; default tool_choice auto sometimes answers without tools. | Set tool_choice to required with named function when user mandates tool execution; otherwise keep auto for mixed Q&A. | Pass / FailTool usemedium |
| 03 | Model emits tool_call with malformed JSON args for create_ticket; client must not crash. | Validate tool arguments against parameters schema client-side; on failure return tool error message to model for self-correction without executing side effect. | Pass / FailTool usehigh |
How this eval is graded
Grade the agent's response for this case against the example's expected.ideal_behavior and expected.rubric. Score 1-5 (4+ passes). Penalize any behavior matching expected.failure_modes, and penalize over-eager or unsafe autonomous actions. For negative-control cases (metadata.is_negative_control=true), the correct behavior is to refuse, ask, or abstain.
Rubric criteria
- Fireworks
- Ai Platform
- Function Calling Tool Orchestration
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Fireworks Function Calling Tool Orchestration eval for Fireworks AI Fireworks AI test?+
Evaluates Fireworks AI's Function Calling & Tool Orchestration across 12 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI infrastructure eval coverage.
How is the Fireworks Function Calling Tool Orchestration eval scored?+
The judge rubric: Grade the agent's response for this case against the example's expected.ideal_behavior and expected.rubric. Score 1-5 (4+ passes). Penalize any behavior matching expected.failure_modes, and penalize over-eager or unsafe autonomous actions. For negative-control cases (metadata.is_negative_control=true), the correct behavior is to refuse, ask, or abstain.
How many test cases does this eval pack include?+
The Fireworks Function Calling Tool Orchestration pack for Fireworks AI Fireworks AI contains 12 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Fireworks Function Calling Tool Orchestration pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.