Eval directory

Evals for Netlify

Eval coverage for Netlify, mapped from its public product surface.

About Netlify

Netlify is a platform for building and shipping web apps, where projects can be started from an AI agent prompt, a Git repository, the CLI, or drag-and-drop, all landing on the same production infrastructure. It bundles agent-ready primitives such as serverless functions, a database, blob storage, edge logic, a global CDN, and AI model access via Agent Runners and AI Gateway. Plans range from a free tier through credit-based Personal and Pro tiers to an Enterprise tier with SLA, SSO, and dedicated support.

Industry

AI-native web development and deployment platform

Website

www.netlify.com

Use the eval library for Netlify

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Netlify?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Project creation and deploy paths

The multiple entry points into a Netlify project — starting from an AI agent prompt, importing a Git repository, deploying from the terminal, or dragging and dropping a folder — and the claim that every path lands on the same project and the same production infrastructure.

“Deploy to a global CDN in seconds, then choose when to make your project public.” www.netlify.com

Mapped capabilities

4 capabilities

Start with an AI agent prompt
Describing what to build from the dashboard and having an agent scaffold and deploy the project.
Deploy from Git
Importing a repository, auto-deploy on every push, and a preview URL for every PR.
Deploy from the CLI
Installing netlify-cli, deploying from a working directory, and the no-login anonymous start path.
Drag and drop (Netlify Drop)
Dropping a project folder to deploy in seconds without a repository or build config.

Previews, privacy, and release control

The prompt–preview–repeat workflow: private preview URLs for in-progress work, new projects private by default until explicitly published, and one-click rollback of any deploy.

“Every change gets a private preview URL before it's live.” www.netlify.com

Mapped capabilities

4 capabilities

Deploy previews and branch deploys
Unlimited previews, per-PR preview URLs, and sharing in-progress work before it reaches production.
Private by default and publishing
New projects staying private until the user chooses to publish, and what changes at publish time.
Rollback and deploy history
Reverting to a prior deploy in one click and reasoning about which version serves the main domain.
Preview Servers
Sharing a customizable preview environment for review ahead of launch.

Illustrative example

Input: I just dragged my project folder into Netlify Drop and it deployed. Is the site publicly accessible to anyone right now?
Expected behavior: Answers no: new projects start private by default and become publicly accessible only when the user explicitly chooses to publish. Points to the publish step as the action that makes it public.

Backend primitives for fullstack apps

The agent-ready primitives an app is assembled from on the same production infrastructure: serverless functions for APIs, Netlify Database for structured data, blob storage for files and images, edge logic, scheduled and background work, and the global CDN.

Mapped capabilities

4 capabilities

Serverless functions and APIs
Building APIs with functions, including the agent-experience-oriented function design.
Netlify Database
Instant structured data storage provisioned for AI-native development workflows.
Blob storage and image optimization
Storing files and images and optimizing images as part of the app.
Edge logic, scheduled jobs, and background work
Running logic at the edge, scheduling jobs, handling background tasks, and global CDN delivery.

AI models, Agent Runners, and AI Gateway

Netlify's AI surface: running coding agents on the platform, routing model calls through AI Gateway, and choosing among commercial and open-weight models (DeepSeek, Qwen, GLM, Kimi) via OpenRouter and OpenCode partnerships.

“Netlify now supports open-weight models like DeepSeek, Qwen, GLM, and Kimi through our partnerships with OpenRouter and OpenCode.” www.netlify.com

Mapped capabilities

4 capabilities

Agent Runners
Running agents on the platform, including OpenCode support, model controls, and Ask mode.
AI Gateway and model choice
Accessing commercial and open-weight models, including OpenRouter support, without leaving the platform.
Adding Netlify to a coding agent
Connecting an external agent (Claude, Codex) to Netlify, including the MCP/stateless-server and netlify.ai paths.
AI features in an app
Adding AI inference to a deployed app, such as a real-time chatbot with no separate backend.

Plans, credits, and billing mechanics

The plan ladder from a free forever tier through credit-based Personal and Pro tiers to Enterprise, and the credit model that meters usage — including per-production-deploy credit cost and variable AI inference cost.

“Production deploys 15 credits each” www.netlify.com

Mapped capabilities

4 capabilities

Plan tiers and what each unlocks
Free, Personal ($9/mo), Pro ($20/mo with unlimited members), and Enterprise, and the feature deltas between them.
Monthly credit allowances
The 300-credit free limit, 1,000 Personal credits, and selectable Pro tiers from 3,000 to 20,000 credits.
What consumes credits
Production deploys at 15 credits each, variable AI inference cost, and what stays unlimited (previews, branch deploys).
Team and Enterprise commercial terms
Unlimited members, private organization repos, concurrent builds, and the Enterprise contact-sales path.

Illustrative example

Input: I'm on Netlify's free plan. How many credits do I get each month, and how many does one production deploy use?
Expected behavior: States the free plan's 300 credit per month limit and that a production deploy costs 15 credits each. Notes that deploy previews and branch deploys are unlimited and do not draw down that allowance.

Security, domains, and operations

The operational envelope around a deployed project: firewall traffic rules and rate limiting, secret detection, custom domains with SSL and Netlify DNS, analytics and observability windows, and Enterprise-grade SLA, SSO/SCIM, and log drains.

Mapped capabilities

4 capabilities

Traffic protection and secrets
Firewall traffic rules, basic rate limiting, smart secret detection, and password protection versus Advanced Security.
Custom domains, SSL, and DNS
Adding custom domains with SSL and managing them through Netlify DNS.
Monitoring, analytics, and log drains
1-day observability on Personal, 30-day analytics on Pro, and Enterprise log drains.
Enterprise governance and support
99.99% SLA, SSO and SCIM, organization management, and 24/7 dedicated support.

Coverage is mapped from Netlify's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Netlify test?+

The coverage map is generated from Netlify's own public product surface (AI-native web development and deployment platform): 6 scoring areas — Project creation and deploy paths, Previews, privacy, and release control, and Backend primitives for fullstack apps, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Netlify evals scored?+

Every case generated for Netlify — across Project creation and deploy paths and Previews, privacy, and release control and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Netlify library include?+

The full Netlify library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Start with an AI agent prompt and Deploy from Git under Project creation and deploy paths); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Netlify or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Netlify areas and set them up in a Corsac workspace, where you can run every test case against Netlify or your own agent with your own data.