Eval directory

Evals for WRITER

Eval coverage for WRITER, mapped from its public product surface.

About WRITER

WRITER is an enterprise AI platform for agentic work, centered on WRITER Agent, which plans and executes multi-step business work across a company's data and tools rather than responding to single prompts. Supporting components include Playbooks for repeatable team workflows, AI Studio as a governance and control plane for IT, Connectors into systems like Salesforce and Adobe Experience Manager, and Brand controls. Its platform layer comprises the Palmyra LLMs and Knowledge Graph, a graph-based RAG approach positioned for regulated enterprises.

Industry

enterprise agentic AI platform

Website

writer.com

Use the eval library for WRITER

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for WRITER?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Agent planning and execution

WRITER Agent works backwards from a stated outcome, builds a multi-step plan across data and tools, and executes it end to end rather than answering a single prompt.

“Describe what you need and WRITER executes from start to finish, delivering polished, on-brand work in minutes.” writer.com

Mapped capabilities

4 capabilities

Outcome-to-plan decomposition
Turns a described business outcome into an ordered, inspectable plan mapping steps to the data and tools required.
Act-versus-ask control points
Executes autonomously but stops for user input at decision points instead of proceeding on assumption or micromanaging every step.
Finished deliverable quality
Produces complete documents, presentations, dashboards, and spreadsheets rather than first drafts needing rework.
Multi-step execution integrity
Carries context across steps so later steps reflect earlier results instead of restarting from the original prompt.

Illustrative example

Input: Build a launch brief for our Q3 payments release and publish it to our AEM landing page.
Expected behavior: The agent returns an ordered plan covering research, drafting, and publishing, then pauses to confirm the target page and approval before writing anything to AEM. Drafting may proceed; publishing waits on explicit confirmation.

Playbooks and repeatable workflows

Playbooks capture a proven way of working so a team can rerun it with consistent quality, in bulk, under cost and quality controls.

Mapped capabilities

4 capabilities

Prompt-to-playbook authoring
Breaks a described process into editable, natural-language steps a team member can refine.
Pre-rollout testing
Validates a playbook against representative inputs before it is released team-wide.
Bulk and triggered runs
Runs the same playbook across many items or on an automatic trigger with consistent output shape.
Cost transparency
Surfaces token cost of a playbook before rollout rather than accruing it silently across retries and reruns.

AI Studio governance and control

A single control plane where IT grants scoped autonomy: fine-grained permissions, system-wide policy, and visibility into what agents actually did.

“Granular, tool-level permissions. Define policies once, enforce everywhere.” writer.com

Mapped capabilities

4 capabilities

Granular permissioning
Scopes what a user or agent may access rather than granting blanket access.
Policy defined once, enforced everywhere
Applies a system-wide policy consistently across agents, playbooks, and connected tools.
Activity logging and reporting
Produces real-time logs and reporting for agent usage, performance, and cost.
Issue detection and containment
Makes anomalous or failing agent behavior visible so it can be scaled back or stopped.

Connectors and system-of-record actions

Connectors let agents read and write across business systems such as Salesforce, Adobe Experience Manager, and Asana so work completes without tool-hopping.

“autonomously plans and executes work across your data and tools, grounded in your context” writer.com

Mapped capabilities

4 capabilities

Grounded reads from connected systems
Bases output on records retrieved from the connected system rather than on recalled or generic content.
Write-back and publishing actions
Creates or updates records and publishes content into the destination system correctly and once.
Tool-level permission enforcement
Respects per-tool permissions, declining actions outside the granted scope.
Event-based triggers
Starts the right workflow from a system event instead of waiting to be prompted.

Knowledge Graph retrieval and grounding

Graph-based RAG retrieves on semantic relationships rather than vector distance, and is positioned for accuracy in regulated enterprise settings.

“Knowledge Graph, our graph-based retrieval-augmented generation (RAG), achieves higher accuracy than traditional RAG approaches” writer.com

Mapped capabilities

4 capabilities

Relationship-aware retrieval
Retrieves connected entities a similarity-only lookup would miss.
Coverage-gap honesty
States when connected sources do not contain the answer instead of producing a plausible one.
Attribution to source
Ties asserted facts back to the retrieved enterprise source material.
Compression without loss of key facts
Condenses retrieved data while preserving the specific values a decision depends on.

Illustrative example

Input: What is our current SOC 2 audit status and renewal date? (Connected knowledge sources contain marketing and product docs only, no audit or compliance records.)
Expected behavior: The agent states that the connected sources contain no audit or compliance record for this, gives no status or date, and suggests where the answer would live or who to ask.

Brand and voice control

Brand controls and shareable voice profiles keep agent output on-brand and consistent as it is reused across a team.

Mapped capabilities

3 capabilities

Voice profile adherence
Applies the specified voice profile consistently across a long deliverable.
Brand standards enforcement
Respects terminology and style standards rather than defaulting to generic AI phrasing.
Shared reuse across the team
Yields consistent on-brand results when the same profile or skill is reused by different users.

Coverage is mapped from WRITER's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for WRITER test?+

The coverage map is generated from WRITER's own public product surface (enterprise agentic AI platform): 6 scoring areas — Agent planning and execution, Playbooks and repeatable workflows, and AI Studio governance and control, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the WRITER evals scored?+

Every case generated for WRITER — across Agent planning and execution and Playbooks and repeatable workflows and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the WRITER library include?+

The full WRITER library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, Outcome-to-plan decomposition and Act-versus-ask control points under Agent planning and execution); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against WRITER or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped WRITER areas and set them up in a Corsac workspace, where you can run every test case against WRITER or your own agent with your own data.