Eval directory · AI Platform

Evals for Modal

Evaluation packs covering adversarial robustness, safety gates, workflow quality, and operator-level checks for Modal AI products.

About Modal

Modal is a serverless cloud platform for running GPU workloads, ML inference, data pipelines, and web apps — all from Python, with no infrastructure to manage. Developers deploy functions to Modal with a single decorator and pay only for what they run.

Employees

~50

Industry

Serverless AI Infrastructure

Headquarters

New York, NY

Website

modal.com

Use the eval library for Modal

All 71 test cases — inputs, expected behavior, and pass/fail checks — runnable in Corsac with your own data.

Related in AI Platform

All evals →

More AI Platform eval libraries

Coverage map

What would you measure for Modal?

7 areas · 71 graded scenarios

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Distributed Dict Queue

Evaluates Modal's Distributed Dict & Queue across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI infrastructure eval coverage.

Mapped capabilities

10 scenarios

modal.Dict put get
modal.Queue producer consumer
Dict partition discovery gap

Public sample case

Input: Fan-out .map workers need modal.Dict().put(job_id, status); coordinator .get(job_id).
Expected behavior: Agent uses modal.Dict with namespaced keys, handles missing key, documents eventual consistency discovery_gap.
Check: Pass / fail check

Function Runtime Cold Start

Evaluates Modal's Function Runtime & Cold Start across 11 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI infrastructure eval coverage.

Mapped capabilities

11 scenarios

GPU type selection
min_containers warm pool
memory_snapshot enablement

Public sample case

Input: Training job uses @app.function(gpu='A10G', memory=32768, timeout=3600) on Image.debian_slim().pip_install('torch'). Logs show CUDA OOM at step 400; model needs ~40GB VRAM. Docs list gpu='A100-80GB' for large fine-tunes.
Expected behavior: Agent updates decorator to gpu='A100-80GB' (or documents trade-off with gradient checkpointing), redeploys via modal deploy, and records GPU choice rationale tied to VRAM estimate.
Check: Pass / fail check

Sandboxes Code Execution

Evaluates Modal's Sandboxes & Code Execution across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI infrastructure eval coverage.

Mapped capabilities

10 scenarios

Sandbox.create app binding
sandbox.exec command
sandbox network egress

Public sample case

Input: Agent code calls modal.Sandbox.create() without app=; fails per docs. Must bind to App for billing and scope.
Expected behavior: Agent passes app=modal.App.lookup('code-runner', create_if_missing=True), documents create_if_missing risk, sets timeout on Sandbox.
Check: Pass / fail check

Scheduled Jobs Cron

Evaluates Modal's Scheduled Jobs & Cron across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI infrastructure eval coverage.

Mapped capabilities

9 scenarios

modal.Period schedule
cron expression
overlapping runs

Secrets Billing Observability

Evaluates Modal's Secrets, Billing & Observability across 11 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI infrastructure eval coverage.

Mapped capabilities

11 scenarios

Secret.from_name injection
App-level secrets scope
App.set_tags billing

Volumes Image Build Cache

Evaluates Modal's Volumes & Image Build Cache across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI infrastructure eval coverage.

Mapped capabilities

10 scenarios

volume.commit before read
concurrent writers
volume.reload staleness

Web Endpoints Request Auth

Evaluates Modal's Web Endpoints & Request Auth across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI infrastructure eval coverage.

Mapped capabilities

10 scenarios

web_endpoint decorator
fastapi_endpoint mounting
requires_proxy_auth

Frequently asked questions

What do the Corsac evals for Modal test?+

Each eval pack tests Modal's public product surface — including Distributed Dict Queue, Function Runtime Cold Start, and Sandboxes Code Execution — against graded scenarios covering adversarial robustness, workflow quality, safety gates, and operator quality. Every pack is runnable in Corsac with your own data.

How are the Modal evals scored?+

Pass/fail checks plus an LLM judge scoring 1–5 against each of the 71 Modal cases — from Function Runtime Cold Start (11 scenarios) down to the smallest pack — its own expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published to the Modal library.

How many test cases does the Modal library include?+

The Modal eval library includes 71 graded test cases across 7 eval packs, the largest being Function Runtime Cold Start with 11 scenarios. Each case defines an input, expected behavior, and pass/fail criteria.

How do I run these evals against Modal or my own agent?+

Request the library with your work email above and we'll set it up in a Corsac workspace, where you can run all 7 Modal packs — Distributed Dict Queue and Function Runtime Cold Start and the rest — against Modal or your own agent with your own data.