Air For Privilege Clawback Prevention And Privilege Log Generation

How this eval is graded

Pass/fail and graded 1–5 by an LLM judge, depending on the test.

Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.

• C-001PASS if the agent's final classification of the document is NOT PRIVILEGED. FAIL if the final classification is PRIVILEGED, PRIVILEGED-CONTESTED, partially privileged, or any label other than not privileged.
• C-002PASS if the rationale explicitly notes that James Whitmore, although he holds a J.D. (and/or formerly served as in-house counsel), is currently acting in a non-legal business capacity as VP of Corporate Strategy. FAIL if the rationale never addresses the distinction between the sender's legal credential and his current non-legal role.
• C-003PASS if the rationale references the email body's explicit statement that this is 'a strategic call, not a legal one' (or equivalent paraphrase) as evidence the sender disclaimed a legal advisory role. FAIL if the rationale makes no mention of the sender's in-document disclaimer.
• C-004PASS if the rationale states that the communication's substance is business/competitive strategy and contains no legal question, legal analysis, or legal advice. FAIL if the rationale does not characterize the substantive content of the email or does not address the absence of legal advice.
• C-005PASS if the rationale concludes that no attorney-client relationship or attorney-client privileged communication is invoked or established by this email. FAIL if the rationale omits any conclusion about whether an attorney-client relationship/communication exists.
• C-006 · NEGFAIL if the agent classifies the document as PRIVILEGED with reasoning that relies primarily or solely on the sender's J.D. credential, his former counsel role, or his company email address. PASS if the agent does not treat the J.D. signature credential or sender identity alone as grounds for privilege.
• C-007 · NEGFAIL if the agent assigns a PRIVILEGED-CONTESTED, borderline, or 'needs attorney review' status, or otherwise escalates/defers rather than issuing a clear not-privileged determination on this unambiguous non-legal communication. PASS if the agent issues a definitive not-privileged classification without escalation.
• C-008 · NEGFAIL if the agent generates a privilege log entry for this document (e.g., an entry describing a 'communication from company attorney regarding corporate strategy' or any other log entry). PASS if no privilege log entry is produced, consistent with a not-privileged determination.
• C-009 · NEGFAIL if the agent acknowledges the sender's explicit disclaimer of a legal role but nonetheless classifies the document as PRIVILEGED (or recommends treating it as privileged) 'out of caution', 'to be safe', or similar over-inclusive reasoning. PASS if the agent's acknowledged evidence and final classification are consistent, with no precautionary over-designation.
• C-010 · NEGFAIL if the rationale asserts unstated facts to justify the classification, such as claiming Whitmore still serves in the legal department, that the CEO sought legal advice, or that the email references litigation or counsel involvement not present in the text. PASS if the rationale relies only on facts present in the provided document and scenario.

Works with

Relativity

Frequently asked questions