Streaming, and the 100-second edge

A request that does not stream sends you nothing at all until the whole answer exists. For a one-line reply that is invisible; for a long generation it is the entire problem. Minutes can pass between the moment your request reaches the model and the moment the first byte of the answer would come back, and the CDN edge in front of this site allows roughly 100 seconds of silence before it gives up on a response. Past that the request dies at the edge while the model is still writing. One field prevents it, and nothing about what you are charged changes when you set it.

Why a long buffered request dies

A buffered request is one round trip with a long pause in the middle. Your client sends the body, the model writes the entire answer, and only when the last token is finished do the first bytes come back down the connection. Between those two moments the connection carries nothing at all, and the edge is watching exactly that: roughly 100 seconds of silence is all it allows before it decides the response is not coming. What you see is a failed request. What happened is that the answer was still being written.

A streamed request has no pause to run out. Bytes leave for your client with the first token, and every token after it arrives as it is produced, so the connection is never silent long enough for the edge to give up on it. That is the part worth keeping: this is not a longer timeout, it is a response that never goes quiet. A generation that takes four minutes streams for four minutes, and arrives.

The fix is one field

Streaming here is not a different endpoint, a different key or a different price. It is one field in the body you are already sending, stream: true, and it works the same way on /v1/chat/completions and on /v1/responses.

The official SDKs spell the same field their own way: stream=True in Python, stream: true in Node. What changes on your side is only the shape of the answer — chunks you iterate over as they arrive, rather than one object you wait for — and every client library ships that loop already written.

What the meter does around a stream

Streaming does not put you outside the meter, and it does not hand you a bill you cannot check. The usage still arrives: the final frame of the stream carries the token counts, and on chat completions the gateway sets stream_options.include_usage itself, so it is not something you can forget to ask for. The meter settles on the same numbers whether the answer was streamed or buffered.

Around that, the sequence is the one every request here follows. A hold is taken when the request starts — your portal balance moves then, not when the last token lands — and the settlement happens on the usage the stream's final frame reports, at the prices recorded when the hold was taken. If the connection drops mid-stream, the hold is reconciled and released rather than billed: tokens that never reached a settlement are never charged.

Three clocks, not one

When a request feels slow, the portal will tell you which kind of slow it is. Three clocks are kept for every request — upload, time to first token, and total — because those are three different problems wearing one word. A long upload is your own uplink handing us the body, and nothing on this side will shorten it. A short time to first token under a long total is a long generation doing exactly what it should, which is the case streaming exists for. A long time to first token is the one worth writing to us about.

One more number a client should be reading. When a request is refused with 429, the Retry-After header passes through untouched — it is the header the official SDKs pace their retries by, so a client that respects it waits the interval we actually sent instead of guessing at one.