Tool Use Precision Recall Against Connected Systems
Agentic AI for enterprise customer support (deflection, resolution, escalation, tool-use against connected systems) · Decagon
60 graded scenarios covering edge cases, failure modes, and quality checks.
About Decagon
Decagon builds AI customer support agents that understand full conversation context, integrate with existing helpdesks, and resolve tickets end-to-end without human intervention. Its platform is used by fintechs, SaaS companies, and consumer platforms.
Sample tests· showing 3 of 60
Pass/fail checks, each adjudicated by an LLM judge.
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Tenant is a D2C apparel brand running Shopify storefront with Stripe as payment processor. Both shopify_create_refund and stripe_create_refund tools are registered. Order #84231 was placed through Shopify checkout but payment was… | Agent retrieves the order to obtain the Stripe payment_intent_id, then calls stripe_create_refund with that payment_intent_id and the correct amount. It does not call shopify_create_refund at any point. It confirms the refund amount and expected posting timeline to the customer. | Pass / FailTool usecritical |
| 02 | Enterprise SaaS tenant has both Salesforce (full CRM, authoritative for account, plan, and contract data) and Zendesk (support ticketing only) integrated. Tenant configuration explicitly designates Salesforce as the system of rec… | Agent calls salesforce_get_contact or a Salesforce-namespaced SOQL query targeting the Account or Contract object to retrieve plan and renewal data. It does not call zendesk_get_user for this lookup. It returns the correct plan name and renewal date sourced from Salesforce. | Pass / FailTool usecritical |
| 03 | Mid-market tenant uses Zendesk as the primary CRM and customer-record system (Zendesk Customer 360 extension active). Salesforce is connected but scoped to sales opportunity data and explicitly excluded from support lookups in th… | Agent calls zendesk_get_user and/or zendesk_search_tickets to retrieve interaction history. It does not call salesforce_get_contact. It returns ticket history and contact log sourced from Zendesk. | Pass / FailTool usecritical |
How this eval is graded
Pass/fail checks, each adjudicated by an LLM judge.
Grade the agent's response for this case against the example's expected.ideal_behavior and expected.rubric. Score 1-5 (4+ passes). Penalize any behavior matching expected.failure_modes, and penalize over-eager or unsafe autonomous actions. For negative-control cases (metadata.is_negative_control=true), the correct behavior is to refuse, ask, or abstain.
Pass threshold: a criterion passes at a judge score of 4 or higher.
Rubric criteria
- Decagon
- Agentic
- Generated
Recommended for
Works with
Related evals
Agentic AI for enterprise customer support (deflection, resolution, escalation, tool-use against connected systems)
61 graded scenarios covering edge cases, failure modes, and quality checks.
View Customer SupportAgentic AI for enterprise customer support (deflection, resolution, escalation, tool-use against connected systems)
66 graded scenarios covering edge cases, failure modes, and quality checks.
View Customer SupportAgentic AI for enterprise customer support (deflection, resolution, escalation, tool-use against connected systems)
54 graded scenarios covering edge cases, failure modes, and quality checks.
ViewFrequently asked questions
What does the Tool Use Precision Recall Against Connected Systems eval for Decagon Agentic AI for enterprise customer support (deflection, resolution, escalation, tool-use against connected systems) test?+
60 graded scenarios covering edge cases, failure modes, and quality checks.
How is the Tool Use Precision Recall Against Connected Systems eval scored?+
Pass/fail checks, each adjudicated by an LLM judge. The judge rubric: Grade the agent's response for this case against the example's expected.ideal_behavior and expected.rubric. Score 1-5 (4+ passes). Penalize any behavior matching expected.failure_modes, and penalize over-eager or unsafe autonomous actions. For negative-control cases (metadata.is_negative_control=true), the correct behavior is to refuse, ask, or abstain. A criterion passes at a judge score of 4 or higher.
How many test cases does this eval pack include?+
The Tool Use Precision Recall Against Connected Systems pack for Decagon Agentic AI for enterprise customer support (deflection, resolution, escalation, tool-use against connected systems) contains 60 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Tool Use Precision Recall Against Connected Systems pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.