Eval directory

Evals for Snowflake

Eval coverage for Snowflake, mapped from its public product surface.

About Snowflake

Snowflake is a fully managed cloud platform that unifies data ingestion, processing, analytics, transactions, and AI/ML workloads in one governed environment. It offers AI services such as Cortex AI, Cortex Analyst and Cortex Search, plus agent products CoCo (a data-native AI coding agent available as desktop, CLI, and in-browser Snowsight) and CoWork. It is sold on a consumption-based pricing model across tiered editions (Standard, Enterprise, Business Critical, Virtual Private Snowflake).

Industry

enterprise cloud data platform with AI/agent services

Website

www.snowflake.com

Use the eval library for Snowflake

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Snowflake?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Conversational Data Analytics

Natural-language access to governed enterprise data through Cortex Analyst, Cortex Search, and the CoWork ask-anything surface, including retrieval over structured and unstructured sources.

“Innovate using sensitive data within a governed perimeter. Ensure your proprietary information never leaves Snowflake.” www.snowflake.com

Mapped capabilities

4 capabilities

Natural-language question to SQL/analytics answer
Turning a business question into a correct, grounded analytical result via Cortex Analyst.
Retrieval over unstructured and multimodal data
Cortex Search-backed lookup across documents and mixed data types.
Grounded answers with source attribution
Answers tied to enterprise data rather than unsupported model recall.
Data agent construction for conversational apps
Assembling agents that combine structured and unstructured retrieval.

Agentic Coding and Workflow Automation (CoCo)

The data-native coding agent across its shipped surfaces — Desktop (macOS/Windows), CLI, and in-browser Snowsight — including pipeline and migration work.

Mapped capabilities

4 capabilities

Surface selection and install guidance
Choosing among Desktop builds, CLI install, and Snowsight browser use for a given environment.
AI-assisted pipeline and warehouse migration
Legacy ETL modernization workflows aligned with the AIM migration suite.
Local file and terminal tool use
Desktop integration with local files, terminals, and the connected Snowflake account.
Scoping and confirmation before acting on data
Stating intended changes and their blast radius before executing them.

Illustrative example

Input: My laptop is Windows on ARM and IT blocks software installs. How can I use Snowflake CoCo today?
Expected behavior: Leads with the in-browser Snowsight option, which requires a Snowflake account and no install. Mentions the Windows ARM desktop download and the PowerShell CLI installer as alternatives if the install restriction is lifted.

Governance, Security and Data Perimeter

Horizon Catalog and Horizon Context controls that keep data and AI assets governed, including edition-gated security features and cross-engine access to open table formats.

“Security with automatic encryption of all data” www.snowflake.com

Mapped capabilities

4 capabilities

Granular governance and privacy controls
Access, masking, and policy behavior described for Enterprise and above.
Governed AI perimeter claims
Accurate handling of the "data never leaves Snowflake" positioning.
Advanced security tiers
Tri-Secret Secure, private connectivity, and VPS isolation.
Cross-engine and Iceberg access control
Read/write governance for open table formats across engines.

Pricing, Editions and FinOps

The consumption-based pricing model, the four editions and their feature boundaries, and cost/performance optimization guidance for buyers and platform owners.

“We keep pricing simple with a consumption-based pricing model.” www.snowflake.com

Mapped capabilities

4 capabilities

Edition-to-feature mapping
Which capabilities require Standard, Enterprise, Business Critical, or VPS.
Consumption model explanation
Describing consumption-based billing without inventing rates.
Routing to pricing calculator and consumption table
Directing platform- and region-specific cost questions to the correct resource.
Cost and performance optimization guidance
Reducing spend within the documented optimization surface.

Illustrative example

Input: We're on Standard edition and our auditor wants cross-region failover and failback for disaster recovery. Can we turn that on, or do we need to change editions?
Expected behavior: Explains that failover and failback are Business Critical features, not part of Standard or Enterprise, and identifies Business Critical or VPS as the upgrade path. Points cost questions to the consumption table or pricing calculator instead of quoting a rate.

Unified Data and Transactional Platform

Ingestion, transactional workloads, and ecosystem interoperability: Snowflake Postgres GA, Unistore, Openflow, Notebooks, Adaptive Compute, Marketplace and data sharing.

Mapped capabilities

4 capabilities

Snowflake Postgres adoption and migration
Running production Postgres alongside analytics with zero code changes.
Hybrid transactional/analytical queries
Joining Unistore transactional data with historical data in one SQL statement.
Ingestion and pipeline setup
Openflow-based ingestion in place of brittle external ETL.
Ecosystem sharing and interoperability
Marketplace, data sharing, and open table format interop across clouds and regions.

Continuity, Recovery and Observability

Business continuity and operational visibility: Time Travel, failover/failback, backups and uptime commitments, and Snowflake Trail observability.

“Virtual Private Snowflake (VPS) includes all the features of Business Critical Edition, but in a completely separate Snowflake environment” www.snowflake.com

Mapped capabilities

4 capabilities

Time Travel and extended retention windows
Recovery of prior data states and edition-dependent window length.
Failover, failback and disaster recovery
Cross-region continuity behavior and its edition requirements.
Backup and availability commitments
Postgres continuous backups and stated uptime SLA, reported accurately.
Workload observability
Snowflake Trail for monitoring and diagnosing running workloads.

Coverage is mapped from Snowflake's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Snowflake test?+

The coverage map is generated from Snowflake's own public product surface (enterprise cloud data platform with AI/agent services): 6 scoring areas — Conversational Data Analytics, Agentic Coding and Workflow Automation (CoCo), and Governance, Security and Data Perimeter, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Snowflake evals scored?+

Every case generated for Snowflake — across Conversational Data Analytics and Agentic Coding and Workflow Automation (CoCo) and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Snowflake library include?+

The full Snowflake library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Natural-language question to SQL/analytics answer and Retrieval over unstructured and multimodal data under Conversational Data Analytics); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Snowflake or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Snowflake areas and set them up in a Corsac workspace, where you can run every test case against Snowflake or your own agent with your own data.