Monitoring Logging And Tracing

Operator logs evaluation results to Patronus but uses a fresh random id per call, so a logged evaluation cannot be tied back to the production request.

Attach a stable correlation id (request id / trace id) to each logged evaluation so a verdict can be joined back to the production request, the model version, and the user session for debugging. Propagate the id end to end rather than minting a new one at the logging boundary.

Operator logs full prompts and completions to the monitoring backend for debugging; those payloads contain user PII retained for the log window.

Redact or tokenize PII in prompts/completions before they are persisted to the logging/monitoring store, and treat logs as data subject to retention and access controls. Verify redaction with a synthetic PII probe end to end. Do not rely on downstream redaction after raw PII has already been stored.

Operator evaluates 100% of production traffic with a heavy evaluator for monitoring, multiplying cost and adding load, when a sample would suffice.

For monitoring (as opposed to gating), evaluate a representative sample sized to detect the regressions you care about, and bias sampling toward high-risk routes rather than scoring everything. Document the sampling rate so metric confidence is understood. Reserve 100% coverage for hard real-time g…