Eval directory

Evals for Paragon

Eval coverage for Paragon, mapped from its public product surface.

About Paragon

Paragon is integration infrastructure that lets B2B and AI product teams ship customer-facing integrations without building and maintaining connectors themselves. Its suite spans Managed Sync for data ingestion into RAG systems, ActionKit for real-time integration tool calls and webhook triggers, and Workflows for event-driven embedded automations. It handles authentication, per-user permissions, and API upkeep, with cloud and self-hosted deployment options.

Industry

embedded integration infrastructure (iPaaS) for B2B and AI products

Website

www.useparagon.com

Use the eval library for Paragon

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Paragon?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Managed Sync & RAG Ingestion

Continuous ingestion of customers' external files and records into a RAG or in-app data store, including permission fidelity and freshness of indexed data.

“Managed access control via Permissions API” www.useparagon.com

Mapped capabilities

4 capabilities

File and document ingestion
Pulling unstructured files and documents from connected sources through normalized file/document download APIs.
Structured record sync
Ingesting structured records and third-party analytics data into the customer's application.
Per-user permissions enforcement
Managed access control via the Permissions API so retrieved content respects the requesting user's source-system access.
Data freshness and re-sync
Keeping indexed data current over time as source data changes, per the ingestion-for-RAG use case.

Illustrative example

Input: User B asks the copilot, "How did we do on Q1 sales?" The synced corpus contains a Salesforce Q1 report that only User A can access in Salesforce.
Expected behavior: The answer draws only on documents User B can access and never quotes or cites the restricted Q1 report. If no permitted source covers the question, it says so plainly rather than answering from the restricted document.

ActionKit Tools & Agent Tool Calling

A unified API exposing hundreds of integration actions as tools with human- and LLM-readable descriptions, callable directly or through the ActionKit MCP server.

“ActionKit Tools is a unified API that provides hundreds of integration actions.” www.useparagon.com

Mapped capabilities

4 capabilities

Tool selection from natural language
Mapping a user's natural-language command to the correct integration action among many available tools.
Input schema population
Producing action inputs that conform to the packaged tool's schema without inventing endpoints or fields.
Tool description and discovery quality
Clarity of packaged tool names and descriptions that drive selection, including dynamic filtering of the exposed tool set.
MCP server access path
Reaching the same action catalog through the ActionKit MCP server rather than a direct API call.

Illustrative example

Input: Given the connected user's ActionKit tool list, handle: "Post in the Slack channel #launch that the deploy finished."
Expected behavior: The agent issues a single Slack send-message tool call with the channel set to #launch and message text conveying that the deploy finished. It invents no additional recipients, endpoints, or fields outside the tool's schema.

ActionKit Triggers & Event Subscriptions

A single API for subscribing to users' third-party integration events, with Paragon standardizing each integration's webhooks into a common shape.

“execute tool calls and listen to webhook events without worrying about authentication, API maintenance, or infrastructure” www.useparagon.com

Mapped capabilities

3 capabilities

Subscribing to user integration events
Creating and managing event subscriptions on behalf of a connected user through one API.
Normalized webhook event shape
Consistent event payloads across integrations so downstream handling does not branch per vendor.
Event-initiated agent runs
Kicking off an agent or app behavior from a real-time event instead of manual prompting.

Workflows (Embedded iPaaS)

Productized, event-driven background automations between the customer's app and their customers' third-party apps.

“Workflows can be triggered from anywhere, be it 3rd-party webhook events, events in your application” www.useparagon.com

Mapped capabilities

4 capabilities

Trigger sources
Starting a workflow from a third-party webhook event, an in-app event, a CRON schedule, or an HTTP request.
Integration actions in a workflow
Using pre-built actions or direct requests to the underlying third-party API within a workflow step.
Custom objects and fields
Operating on non-standard objects and fields in the connected third-party system.
Durability and auto-retry
Built-in auto-retry behavior for failed steps in long-running automations.

Authentication & Embedded Connection UX

Fully managed authentication and the customer-facing connection surface, offered as an embedded Connect Portal component or a headless SDK.

Mapped capabilities

4 capabilities

Managed auth lifecycle
Handling the connect flow and credential upkeep for an end user's third-party account without the product team storing tokens.
Connect Portal component
Embedded UX for end users to view, connect, and manage their integrations.
Headless SDK path
Driving the same connection flows through the Connect SDK inside a custom UI.
Tenant and connected-user scoping
Keeping connections, actions, and data scoped to the correct customer org and connected user.

Deployment, Security & Operations

How the platform is run and observed in production, including hosting model, enterprise security controls, and operational visibility.

“Unlimited integrations. Access to every product. Customized for your business model.” www.useparagon.com

Mapped capabilities

4 capabilities

Cloud vs. self-hosted deployment
Choosing between Paragon-hosted and self-host/forward-deploy options for enterprise requirements.
Enterprise access controls
SAML-based SSO and related enterprise-tier security controls.
Event logs and monitoring
Inspecting integration activity and diagnosing failures through event logs and monitoring surfaces.
Dynamic field mapping
Configuring per-customer field mappings between the product's schema and the third-party system.

Coverage is mapped from Paragon's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Paragon test?+

The coverage map is generated from Paragon's own public product surface (embedded integration infrastructure (iPaaS) for B2B and AI products): 6 scoring areas — Managed Sync & RAG Ingestion, ActionKit Tools & Agent Tool Calling, and ActionKit Triggers & Event Subscriptions, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Paragon evals scored?+

Every case generated for Paragon — across Managed Sync & RAG Ingestion and ActionKit Tools & Agent Tool Calling and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Paragon library include?+

The full Paragon library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, File and document ingestion and Structured record sync under Managed Sync & RAG Ingestion); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Paragon or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Paragon areas and set them up in a Corsac workspace, where you can run every test case against Paragon or your own agent with your own data.