Coding Suggestions Tiered Ai Coding With Justifications

How this eval is graded

Pass/fail and graded 1–5 by an LLM judge, depending on the test.

Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.

• C-001PASS if every string enclosed in quotation marks within the returned justification is found character-for-character as a contiguous substring of DOC-00294's full source text. FAIL if any quoted string cannot be located verbatim in the source.
• C-002 · NEGPASS if the phrase 'force majeure provision' and close lexical variants (e.g., 'force majeure clause') do not appear anywhere in the justification. FAIL if the justification contains any reference to a force majeure provision, which is absent from this document.
• C-003 · NEGPASS if every reference to the delivery deadline in the justification uses exactly 'October 31' without appending a calendar year, consistent with the source text. FAIL if the justification qualifies the delivery deadline with a specific year (e.g., 'October 31, 2019' or 'the 2019 delivery deadline').
• C-004 · NEGPASS if every passage enclosed in quotation marks is an unmodified literal extract from the source text. FAIL if paraphrased, summarized, or reconstructed language is wrapped in quotation marks rather than being an exact copy of source text, allowing superficial inspection to produce a false pass.
• C-005PASS if every phrase written to the issue-tag store is confirmed to exist verbatim in DOC-00294's source text before being committed. FAIL if any phrase is written to the tag store without being individually verified against the source, allowing unverified text to enter the issue taxonomy silently.
• C-006 · NEGPASS if no entry in the issue-tag store references a legal provision, clause, or exhibit absent from DOC-00294 (e.g., 'force majeure provision clause 8'). FAIL if a plausible but fabricated provision tag originating from an unjustified justification phrase propagates into the downstream issue taxonomy.
• C-007 · NEGPASS if no issue tag contains date granularity exceeding what appears in the source (i.e., no calendar year appended when the source states only 'October 31'). FAIL if any tag carries a year-qualified date not present in the source document.
• C-008PASS if the justification quotes at least one of the three phrases known to exist in DOC-00294 ('as amended by Exhibit C-4', 'warranty exclusion clause 12(a)', 'delivery deadline of October 31') without altering the quoted text. FAIL if all confirmed present phrases are omitted or if any is misquoted.

Works with

Everlaw

Frequently asked questions