Eval directory · Content & Writing
Evaluation packs covering adversarial robustness, safety gates, workflow quality, and operator-level checks for Notion AI products.
Notion is an all-in-one workspace for notes, docs, databases, and project management, used by millions of teams worldwide. Its AI layer assists with writing, summarization, action extraction, and knowledge retrieval directly within the workspace.
Use the eval library for Notion
All 4 test cases — inputs, expected behavior, and pass/fail checks — runnable in Corsac with your own data.
Coverage map
1 area · 4 graded scenarios
Every eval set is graded on
Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls
01
Smoke test for writing assistants: instruction-following and safety.
Mapped capabilities
4 scenarios
Example criterion: Notion consistently produces instruction-aligned, high-quality writing while maintaining safety boundaries.
Each eval pack tests Notion's public product surface — including Ai Writing Smoke V1 — against graded scenarios covering adversarial robustness, workflow quality, safety gates, and operator quality. Every pack is runnable in Corsac with your own data.
Every eval set is graded the same way: pass/fail checks plus an LLM judge scoring 1–5 against each case's expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.
The Notion eval library includes 4 graded test cases across 1 eval pack. Each case defines an input, expected behavior, and pass/fail criteria.
Request the library with your work email above and we'll set it up in a Corsac workspace, where you can run every test case against Notion or your own agent with your own data.
