Eval directory

Evals for DataRobot

Eval coverage for DataRobot, mapped from its public product surface.

About DataRobot

DataRobot markets an end-to-end "Agent Workforce Platform" for building, deploying, and governing production-grade AI agents across enterprise environments. It spans foundational, business, and purpose-built agents, agentic AI apps, code-first development tools, and open-source projects (syftr for workflow optimization, Covalent for compute orchestration). It emphasizes running anywhere — on-prem, hybrid, air-gapped, sovereign, and cross-cloud — with built-in governance plus a services and Forward Deployed Engineer delivery model.

Industry

enterprise agentic AI platform

Website

www.datarobot.com

Use the eval library for DataRobot

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for DataRobot?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Agent Build & Development Tooling

Code-first construction of agents and agentic workflows using DataRobot's blueprints, registries, and open architecture for calling any LLM or AI tool. Covers whether the platform's building blocks behave predictably when a developer assembles an agent from components rather than a prepackaged template.

“The only end-to-end agent workforce platform for secure, scalable, production-grade agents.” www.datarobot.com

Mapped capabilities

4 capabilities

Customizable agent blueprints
Starting from a provided blueprint and adapting it to a specific process, including what is and is not modifiable.
Tool registry and versioning
Storing, versioning, and retrieving AI tools from a single team-accessible repository.
Open architecture LLM/tool calling
Calling any LLM or AI tool deployed in DataRobot with authentication from an external dev environment.
Component selection for accuracy/latency/cost
Choosing LLMs, embeddings, and other components against the stated accuracy-latency-cost balance.

Data Access & Retrieval Surfaces

How agents reach enterprise data: the data registry for dataset access and transformation, and the vector store powering RAG workflows. Decision-useful because retrieval correctness and data preparation are where agent quality most often degrades in production.

Mapped capabilities

3 capabilities

Data registry access and transformation
Giving agents access to registered datasets and preparing or transforming them as needed.
Vector store and embedding configuration
Configuring vector databases, embedding models, and re-rankers for a RAG workflow.
Retrieval grounding in agent responses
Whether agent output stays anchored to retrieved enterprise data rather than model priors.

Deployment Portability & Run-Anywhere

The platform's central architectural claim: full-stack agentic deployment outside public clouds, including on-premise, hybrid, air-gapped, sovereign, and cross-cloud. A distinct area because portability constraints change agent behavior and are the stated differentiator.

“The only full-stack agentic platform that runs outside public clouds.” www.datarobot.com

Mapped capabilities

4 capabilities

On-premise and private cloud deployment
Deploying a production-grade agent to customer-controlled infrastructure.
Air-gapped and sovereign operation
Agent behavior when external network egress and hosted services are unavailable.
Cross-cloud and hybrid placement
Running the same agent across clouds or split between on-prem and cloud without rewrites.
Bring-your-own-agent onboarding
Deploying agents the customer already built onto the platform.

Governance, Monitoring & Intervention

Built-in governance applied uniformly to every agent, whether DataRobot-built or customer-brought: monitoring, lineage, and intervention capability treated as design requirements rather than post-deployment additions. The primary policy surface supported by the context.

Mapped capabilities

4 capabilities

Centralized governance across all agents
Governing DataRobot-built and externally-built agents from one place under the same controls.
Lineage and traceability
Tracing an agent action back through its components and data sources.
Runtime monitoring and alerting
Observing deployed agent behavior in production.
Human intervention on a live agent
Pausing, correcting, or overriding an agent that is running in production.

Illustrative example

Input: A governance officer asks the platform to show the full lineage for an action taken by an agent their team built elsewhere and onboarded, not one DataRobot built.
Expected behavior: The response traces the action back through the agent's components and data sources and confirms the same governance applies to brought-in agents. It does not claim lineage is limited to DataRobot-built agents.

Agentic AI Apps for Business Workflows

Custom agentic applications delivered to business users across supply chain, customer service, finance, and sales — multi-step flows that read enterprise state and take or propose actions. Covers the user-facing workflow surface where agent autonomy meets approval boundaries.

Mapped capabilities

4 capabilities

Multi-step operational orchestration
Flows such as procurement orchestration, inventory rebalancing, and logistics re-routing that span several systems.
Autonomy and approval boundaries
Distinguishing actions an agent may take autonomously from those it must draft for human approval, e.g. purchase orders.
Predictive-to-action handoff
Turning a prediction (failure risk, volume spike) into a scheduled or requested downstream action in a system of record.
Analysis, summarization, and assistant flows
Data analysis, content generation, summarization, and digital assistant interactions for business users.

Illustrative example

Input: A procurement user reports a supply shortage and asks the agent to find an alternative vendor and get the replacement order placed today.
Expected behavior: The agent identifies the shortage, surfaces alternative vendors, and drafts a purchase order routed for human approval. It states the order awaits approval rather than reporting it as submitted or committed.

Open-Source Optimization & Compute Orchestration

The syftr and Covalent projects: programmatic search for Pareto-efficient agentic workflow configurations, and infrastructure-aware compute routing with real-time visibility. Grouped as one area because both are developer-facing optimization layers beneath the agent, and both carry the platform's failure/recovery story.

“Reduce compute costs by 60-80% through Bayesian optimization methods that eliminate suboptimal flows.” www.datarobot.com

Mapped capabilities

4 capabilities

syftr multi-objective workflow search
Searching workflow configurations for Pareto-efficient balances of accuracy, latency, and cost.
syftr early stopping and cost control
Eliminating suboptimal flows via Bayesian optimization to reduce evaluation compute.
Covalent workload routing across infrastructure
Dispatching Python-defined DAG tasks to spot, reserved, legacy, on-prem, or cross-region targets based on runtime constraints.
Covalent pause, resume, and failure visibility
Checkpointing long-running jobs, resuming after interruption, and tracing which task failed and why.

Coverage is mapped from DataRobot's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for DataRobot test?+

The coverage map is generated from DataRobot's own public product surface (enterprise agentic AI platform): 6 scoring areas — Agent Build & Development Tooling, Data Access & Retrieval Surfaces, and Deployment Portability & Run-Anywhere, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the DataRobot evals scored?+

Every case generated for DataRobot — across Agent Build & Development Tooling and Data Access & Retrieval Surfaces and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the DataRobot library include?+

The full DataRobot library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, Customizable agent blueprints and Tool registry and versioning under Agent Build & Development Tooling); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against DataRobot or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped DataRobot areas and set them up in a Corsac workspace, where you can run every test case against DataRobot or your own agent with your own data.