Corsac
Use Corsac
All evals
Poolside

Eval directory · Medical & Clinical AI

Evals for Poolside

Evaluation packs covering adversarial robustness, safety gates, workflow quality, and operator-level checks for Poolside AI products.

About Poolside

Poolside develops foundation models and agent systems for software engineering. Its Laguna models support agentic coding, while its platform provides APIs, coding agents, and enterprise administration tools.

Industry

AI Foundation Models for Software Engineering

Website

poolside.ai

Use the eval library for Poolside

All 43 test cases — inputs, expected behavior, and pass/fail checks — runnable in Corsac with your own data.

Generate your own →

Related in Medical & Clinical AI

All evals →

More Medical & Clinical AI eval libraries

Coverage map

What would you measure for Poolside?

2 areas · 43 graded scenarios

Every eval set is graded on

  • Adversarial robustness
  • Workflow quality
  • Safety gates
  • Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

01

Generated Code Correctness Malibu

Mapped capabilities

32 scenarios

  • Single-function generation from NL prompt

Public sample case

Input
Developer is mid-feature in a signup form handler and invokes the IDE plugin's inline generation on a one-line comment above an empty function stub. The generated function will be pasted directly into the form-validation module w…
Expected behavior
Generated function correctly normalizes valid 10-digit US numbers (with or without dashes/parens/spaces, with or without a leading '1' or '+1') to '+1XXXXXXXXXX', and returns null/None or raises (consistently, matching the language's idiom) for: numbers with fewer/more than 10 significant digits af…
Check
Pass / fail check

Public sample case

Input
A CI job runs `pool exec` non-interactively to generate a small date-validation utility for a JS backend, writes it straight to `src/utils/date.js`, and immediately runs the existing Jest suite plus a newly auto-generated smoke t…
Expected behavior
Function correctly implements the full Gregorian leap-year rule: divisible by 4 AND (not divisible by 100 OR divisible by 400). Feb 29 2024 -> true, Feb 29 2023 -> false, Feb 29 1900 -> false, Feb 29 2000 -> true, Feb 30 (any year) -> false, month 13 or month 0 -> false, day 0 -> false.
Check
Pass / fail check

02

Ide Plugin Integration Surface

Mapped capabilities

11 scenarios

  • VS Code marketplace install
  • VS Code minimum-version gate

Public sample case

Input
Agent is instructed to install and activate the Poolside VS Code extension for a new hire's workspace. The extension installs cleanly, but on first activation the entitlement/auth handoff returns a state indicating the user's org…
Expected behavior
Agent installs the extension, attempts first-launch auth handoff once, detects the expired-seat failure state surfaced by the extension/CLI (status output, log line, or notification), stops retrying, and reports back plainly: extension is installed but not activated because the org seat is expired,…
Check
Pass / fail check

Frequently asked questions

What do the Corsac evals for Poolside test?+

Each eval pack tests Poolside's public product surface — including Generated Code Correctness Malibu and Ide Plugin Integration Surface — against graded scenarios covering adversarial robustness, workflow quality, safety gates, and operator quality. Every pack is runnable in Corsac with your own data.

How are the Poolside evals scored?+

Pass/fail checks plus an LLM judge scoring 1–5 against each of the 43 Poolside cases — from Generated Code Correctness Malibu (32 scenarios) down to the smallest pack — its own expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published to the Poolside library.

How many test cases does the Poolside library include?+

The Poolside eval library includes 43 graded test cases across 2 eval packs, the largest being Generated Code Correctness Malibu with 32 scenarios. Each case defines an input, expected behavior, and pass/fail criteria.

How do I run these evals against Poolside or my own agent?+

Request the library with your work email above and we'll set it up in a Corsac workspace, where you can run all 2 Poolside packs — Generated Code Correctness Malibu and Ide Plugin Integration Surface and the rest — against Poolside or your own agent with your own data.