Eval directory

Evals for Neon

Eval coverage for Neon, mapped from its public product surface.

About Neon

Neon is a serverless Postgres backend platform, now part of the Databricks platform, built around Lakebase Postgres. Alongside the database it bundles Managed Better Auth, S3-compatible Object Storage, Functions, a Data API, and an AI Gateway, all branchable per project. It sells on usage-based pricing with a free tier, instant git-like branching, autoscaling, and HIPAA support on its Scale plan.

Industry

serverless Postgres backend platform for apps and AI agents

Website

neon.com

Use the eval library for Neon

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Neon?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Lakebase Postgres core

The serverless Postgres database itself: how compute and storage separate, how it scales, and how data is recovered.

“The backend for apps and agents, built to scale on Lakebase Postgres.” neon.com

Mapped capabilities

4 capabilities

Advanced autoscaling
CPU, memory, and storage scaling to workload; compute sizes and CU ceilings per plan.
Scale to zero
Behavior when a compute is inactive and what that implies for cost and connections.
Instant restore and history window
Configurable history window, snapshots, and scheduled backups as described in pricing.
Connection and project structure
Projects, databases, roles, and where each lives in the Console.

Branching and environment workflows

Git-like branching as the core development workflow, including how non-database services branch alongside Postgres.

Mapped capabilities

4 capabilities

Copy-on-write branch creation
Creating editable copies of a database instantly; root vs child branch storage.
Anonymization
Masking sensitive columns with realistic fake values for safe testing and sharing.
Ephemeral branches
Automatic deletion of obsolete branches after work completes.
Cross-service branch isolation
Auth users and Object Storage namespaces branching with the project.

Auth and Data API

Managed Better Auth and the HTTPS query layer that let an app read and write without custom backend code.

“Data API HTTPS queries with no backend code. Drop-in compatible with Supabase.” neon.com

Mapped capabilities

3 capabilities

Sign-up, OAuth, and sessions
Managed Better Auth flows with users and sessions stored in Postgres.
Users as Postgres data
Querying and joining auth tables that live in the customer's own database.
Data API HTTPS queries
Backend-free queries over HTTPS; documented Supabase drop-in compatibility.

Object Storage and Functions

The beta backend services that sit next to the database: S3-compatible blobs and long-running compute.

“By separating compute and storage, Lakebase Postgres autoscales CPU, memory and storage to fit your workload.” neon.com

Mapped capabilities

4 capabilities

Buckets and access modes
private vs public_read buckets, creation, and how buckets branch.
Objects and presigned URLs
Upload, download, list, delete, and presigned URL generation.
S3 SDK compatibility
AWS SDK for JavaScript, boto3, AWS CLI, and Files SDK pointed at a branch endpoint.
Functions execution model
Long-running Node.js compute without timeouts, deployed alongside the database.

Illustrative example

Input: Our workloads run in AWS eu-central-1. Can we provision Neon Object Storage and Functions in that region right now?
Expected behavior: States that during beta these services run only in AWS US East (Ohio), aws-us-east-2, so eu-central-1 is not available yet, while noting more regions are planned. Does not invent a workaround or an availability date.

Agent and AI surfaces

The paths Neon markets to agents and AI engineering workflows, from one-command setup to model access.

Mapped capabilities

4 capabilities

One-command init
npx neon@latest init creating a project, applying schema, and writing .env.
MCP client connection
Connecting MCP clients to Neon as described on the homepage.
AI Gateway model catalog
One API across frontier and open-weight models, billed on one bill.
Neon credential system
Shared credentials across AI Gateway, Functions, and Object Storage.

Plans, pricing, and compliance

The commercial and regulatory boundaries a buyer must get right: plan tiers, usage rates, and constraints on availability.

“Neon offers HIPAA compliance as a self-serve feature available to customers on the Scale plan.” neon.com

Mapped capabilities

4 capabilities

Plan tiers and limits
Free, Launch, and Scale: project counts, storage, compute size ceilings, metrics retention.
Usage-based rates
Per CU-hr and per GB-month pricing and how typical spend is framed.
HIPAA enablement
Scale-plan self-serve HIPAA, BAA acceptance, org-then-project enablement.
Beta and region constraints
Beta service status and the AWS us-east-2-only availability window.

Illustrative example

Input: We're on the Launch plan and about to store patient records in Neon. Can we switch HIPAA on for our project today?
Expected behavior: Explains HIPAA is self-serve on the Scale plan, so a Launch account must upgrade first. Notes the order of operations: enable HIPAA at the organization level and accept the Business Associate Agreement, then enable it per project.

Coverage is mapped from Neon's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Neon test?+

The coverage map is generated from Neon's own public product surface (serverless Postgres backend platform for apps and AI agents): 6 scoring areas — Lakebase Postgres core, Branching and environment workflows, and Auth and Data API, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Neon evals scored?+

Every case generated for Neon — across Lakebase Postgres core and Branching and environment workflows and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Neon library include?+

The full Neon library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, Advanced autoscaling and Scale to zero under Lakebase Postgres core); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Neon or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Neon areas and set them up in a Corsac workspace, where you can run every test case against Neon or your own agent with your own data.