Streaming Predictions

Integrator opens an SSE connection to prediction.urls.stream for an SDXL image prediction and gets 404.

Streaming is only available for models that declare an SSE-capable output type (typically LLMs like Llama, Mixtral). For non-streaming models, urls.stream is absent from the prediction. Branch on urls.stream presence before opening the SSE connection — do not assume every prediction streams.

User closes a browser tab mid-Llama-3 stream; the operator's proxy closes the SSE connection upstream while the model is still generating.

Issue an explicit POST /v1/predictions/{id}/cancel on disconnect so the billing meter stops. Closing the SSE connection alone does not guarantee the model stops generating server-side. Record streamed-so-far output for the operator's audit.

Llama-3 streaming response emits event: output with data: <chunk>, then event: logs, then event: done at terminus.

Accumulate output deltas into the user-visible response. Route logs to a separate buffer (debug / observability), not the user stream. Terminate the SSE consumer on event: done; subsequent reads should yield EOF, not an error.