Corsac
Use Corsac
All evals
ClickHouse

Eval directory · Data Analysis

Evals for ClickHouse

Evaluation packs covering adversarial robustness, safety gates, workflow quality, and operator-level checks for ClickHouse AI products.

About ClickHouse

ClickHouse is an open-source columnar database management system built for real-time analytics. It processes billions of rows per second with sub-second query latency, making it the database of choice for observability, product analytics, and time-series workloads.

Employees

~500

Industry

Analytical Database

Headquarters

San Francisco, CA

Website

clickhouse.com

Use the eval library for ClickHouse

All 55 test cases — inputs, expected behavior, and pass/fail checks — runnable in Corsac with your own data.

Generate your own →

Related in Data Analysis

All evals →

More Data Analysis eval libraries

Coverage map

What would you measure for ClickHouse?

3 areas · 55 graded scenarios

Every eval set is graded on

  • Adversarial robustness
  • Workflow quality
  • Safety gates
  • Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

01

Async Insert Durability And Ingestion Semantics

Mapped capabilities

23 scenarios

  • async_insert_max_data_size threshold flush trigger

Public sample case

Input
An IoT telemetry pipeline ingests via async_insert into a MergeTree table. Throughput is falling behind during peak load, and the team wants bigger batches with less client-side overhead.
Expected behavior
Agent proposes or applies a concrete higher value, but explicitly states the durability/visibility tradeoff in plain terms: a larger threshold means more rows sit unflushed in server memory longer before a part is written, widening the window of data exposed to loss on a crash and increasing read-y…
Check
Pass / fail check

02

Materialized Views Projections And Ttl Derived Data Consistency

Mapped capabilities

7 scenarios

  • Incremental MV trigger on single-block insert

Public sample case

Input
An engineer just created `events_per_minute_mv`, an incremental materialized view over the `events` source table writing aggregate state rows (countState/sumState) into `events_per_minute_agg` (AggregatingMergeTree). No bulk load…
Expected behavior
Agent inserts one row, queries the target table, and confirms the aggregate value is numerically correct for that single row. It then explicitly states that this single-row/single-block test does NOT validate behavior under bulk inserts that exceed max_insert_block_size or under async_insert batchi…
Check
Pass / fail check

03

Mergetree Engine Correctness Replacingmergetree Aggregatingmergetree Final

Mapped capabilities

25 scenarios

  • ReplacingMergeTree merge-time dedup correctness

Public sample case

Input
A support-tooling agent has HTTP access to a ClickHouse instance backing an order-status service. Table: `CREATE TABLE orders (order_id UInt64, status String, updated_at DateTime, version UInt64) ENGINE = ReplacingMergeTree(versi…
Expected behavior
Agent either runs `SELECT status FROM orders WHERE order_id = 42 ORDER BY version DESC FINAL LIMIT 1` (or an equivalent FINAL/argMax(version) pattern) and returns 'shipped', or checks `system.parts` for unmerged duplicate parts on that key before answering. It does not surface 'pending' as the answ…
Check
Pass / fail check

Frequently asked questions

What do the Corsac evals for ClickHouse test?+

Each eval pack tests ClickHouse's public product surface — including Async Insert Durability And Ingestion Semantics, Materialized Views Projections And Ttl Derived Data Consistency, and Mergetree Engine Correctness Replacingmergetree Aggregatingmergetree Final — against graded scenarios covering adversarial robustness, workflow quality, safety gates, and operator quality. Every pack is runnable in Corsac with your own data.

How are the ClickHouse evals scored?+

Pass/fail checks plus an LLM judge scoring 1–5 against each of the 55 ClickHouse cases — from Mergetree Engine Correctness Replacingmergetree Aggregatingmergetree Final (25 scenarios) down to the smallest pack — its own expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published to the ClickHouse library.

How many test cases does the ClickHouse library include?+

The ClickHouse eval library includes 55 graded test cases across 3 eval packs, the largest being Mergetree Engine Correctness Replacingmergetree Aggregatingmergetree Final with 25 scenarios. Each case defines an input, expected behavior, and pass/fail criteria.

How do I run these evals against ClickHouse or my own agent?+

Request the library with your work email above and we'll set it up in a Corsac workspace, where you can run all 3 ClickHouse packs — Async Insert Durability And Ingestion Semantics and Materialized Views Projections And Ttl Derived Data Consistency and the rest — against ClickHouse or your own agent with your own data.