Streaming responses¶
Subclass BaseStreamMethod. The generic argument is the type of one streamed item: bytes, str, or an event type (see Events).
from aiollm import BaseStreamMethod
class Download(BaseStreamMethod[bytes]): # raw chunks
__request_path__ = "/files/content"
__request_method__ = "GET"
__request_query__ = {"file_id"}
__retryable__ = True
file_id: str
class Generate(BaseStreamMethod[str]): # decoded text pieces
__request_path__ = "/generate"
__request_method__ = "POST"
__request_body_format__ = BodyFormat.JSON
prompt: str
async with session.stream(Download(file_id="f-1")) as stream:
await stream.save("result.bin") # only on Stream[bytes]
async with session.stream(Generate(prompt="Hi")) as stream:
async for piece in stream: # piece: str
print(piece, end="")
- Errors before the stream starts. An error status raises the matching
APIStatusErroras you enter theasync withblock. - Retries. Only opening the stream is retried. A failure while iterating reaches your code, so it never receives duplicated data.
- Cleanup. The connection is released when the block exits, including on
breakor an exception. Iterating the stream after the block, even through an iterator taken inside it, raisesRuntimeError, instead of quietly yielding nothing from a closed connection. - Text decoding. It is incremental, so a multi-byte character split between two chunks still decodes correctly. A charset that is not a text encoding (
base64,zlib) raisesAPIResponseError. - Saving to a file.
Stream.save(path)writes chunks as they arrive, without holding the body in memory, and returns the number of bytes written. It opens, writes and closes the file in a worker thread; a cancelled save closes the file, in the thread too, once the write running there is over. - Timeouts. Streams have no total time limit, only 600 s between two chunks, so a long generation is not cut off. See Timeouts.
- Type safety. Passing a stream method to
send_request, or a buffered method tostream, is a type error. - Format and item type.
BYTESstreams yieldbytesandTEXTstreams yieldstr, so an explicit__response_format__that contradicts the generic argument, such asBaseStreamMethod[str]withBYTES, raisesTypeErrorat class definition, unless the method sets its own__stream_parser__.
Events: SSE and NDJSON¶
For a stream of events, set __response_format__ to ResponseFormat.SSE (text/event-stream) or ResponseFormat.NDJSON (one JSON value per line). The generic argument can be a model or a discriminated union of models, and each event is validated into it.
from typing import Annotated, Literal
from pydantic import Field
from aiollm import ResponseFormat, ServerSentEvent
class TextDelta(BaseAPIType):
type: Literal["text_delta"]
text: str
class MessageStop(BaseAPIType):
type: Literal["message_stop"]
type MessageEvent = Annotated[
TextDelta | MessageStop,
Field(discriminator="type"),
]
class StreamMessage(BaseStreamMethod[MessageEvent]):
__request_path__ = "/messages"
__request_method__ = "POST"
__request_body_format__ = BodyFormat.JSON
__response_format__ = ResponseFormat.SSE
prompt: str
stream: bool = True
class RawEvents(BaseStreamMethod[ServerSentEvent]): # SSE, not validated
__request_path__ = "/events"
__request_method__ = "GET"
async with session.stream(StreamMessage(prompt="Hi")) as events:
async for event in events: # TextDelta | MessageStop
if isinstance(event, TextDelta):
print(event.text, end="")
- Content type. An SSE response must have
Content-Type: text/event-stream; anything else (such as a proxy's HTML login page with status 200) raisesAPIResponseErrorinstead of giving an empty stream. For a server that sends another type, useStreamResponseParser(require_event_stream=False)as the method's__stream_parser__. - Format. It must be set for models and unions: the item type alone doesn't tell SSE from NDJSON. Without it the class definition raises
TypeError, unless the method sets its own__stream_parser__.ServerSentEventitems imply SSE. - Parsing. SSE follows the HTML standard: multi-line
data, comments,event,idandretry, and\n,\r\nor\rline endings. An event split across chunks is held back until it is complete, and an unfinished event at the end of the body is dropped. NDJSON skips blank lines. - Validation. Each event is validated through a
TypeAdapterbuilt once, when the class is defined, from the generic argument as written. That is why discriminated unions andtypealiases work as item types. An event that doesn't match raisesAPIResponseError. - End of stream. By the HTTP and SSE standards, the end of the body is a normal end, so a stream cut between two events looks complete. Only the protocol's own last event proves that it is. Declare it as
__stream_terminal__and the parser checks it: a body that ends without it raisesAPIStreamInterruptedError, and an item after it raisesAPIResponseError(the parser reads the body to its end to see that), an error event included: once the terminal has come, the stream is complete, and anything after it breaks the protocol. Matchers live inaiollm.types:EventTypes("message_stop")matches atype,FieldValues(name, *values)any field,DONEthedata: [DONE]line, anda | beither. They compare values as JSON tells them apart (FieldValues("done", True)does not match{"done": 1}), read a model's fields but never its properties, and are equal when they match the same items (EventTypes("x") == FieldValues("type", "x")). Without a declared terminal (the default),data: [DONE]ends the stream and nothing is checked. Only SSE and NDJSON have events to check, so a terminal on aBYTESorTEXTstream raisesTypeErrorat class definition, unless the method sets its own__stream_parser__.[DONE]and SSE comments are never items. The provider methods declare theirs:response.completed,response.incompleteorresponse.failedfor OpenAI'sStreamResponse,message_stopfor Anthropic'sStreamMessage. - Errors inside the stream. An SSE event whose
event:field iserror(Anthropic) raises, whatever its data, theAPIStatusErrorbuilt by the session's error resolver, and it is never transient: the generation already ran. So does decoded data that the parser'serror_eventsmatcher accepts, also for aServerSentEventitem whose data is JSON (data that is not JSON is yielded as it is). The default,EventTypes("error") | TopLevelError(), accepts JSON whosetypeis"error", or that has notypeand a top-levelerrorthat istrueor a non-empty object or string (OpenAI, Ollama); an event with anothertypemay carry anerrorfield as data (a failed tool call, for example). For other rules, give the method a__stream_parser__ = StreamResponseParser(error_events=...), as OpenAI'sStreamResponsedoes to raiseresponse.failed. When the error event also fits the item type, the validated event is the error'sraw. - Raw events.
BaseStreamMethod[ServerSentEvent]yields events as they are, with theirdata,event,idandretry.[DONE]and error events are handled the same way. - Limits.
StreamResponseParser(max_line_size=...)limits one SSE or NDJSON line andmax_event_size=...the data of one SSE event, in characters; a longer one raisesAPIResponseError. Both are on by default, 32 Mi characters each (MAX_LINE_SIZE,MAX_EVENT_SIZE), since a server that never ends a line would otherwise make the stream grow in memory without bound. They are large because a normal event can be: OpenAI sends a generated image as base64 in one line, and the final event repeats the whole response.Nonelifts a limit; that is your decision to make. Aretryfield with more digits than Python turns into a number (4300 by default) raisesAPIResponseErroras well.
Interrupted streams¶
A connection can break at any point of a long generation. What happens depends on when it breaks:
| When | What you get | Retried |
|---|---|---|
| Before the response arrives: connection refused, DNS, a TLS handshake that breaks off, connect timeout, a certificate that fails verification | APIConnectionError, APITimeoutError or (for the certificate) APITLSError on entering the block |
yes, if the method is retryable and the error is transient |
| The server answers with an error status | the APIStatusError for it, on entering the block |
yes, if the error is transient |
| The connection drops after some items | APIStreamInterruptedError from async for, caused by the APIConnectionError |
no |
No data for 600 s (sock_read) |
APIStreamInterruptedError from async for, caused by the APITimeoutError |
no |
The body ends before the method's __stream_terminal__ |
APIStreamInterruptedError from async for, without a cause |
no |
| The server sends an error event | the APIStatusError for it, from async for |
no |
Items you received before the failure stay yours (items_received counts them), and the connection is released as the block exits, also when the task is cancelled. APIStreamInterruptedError is never transient: trying again means a new request, and the library does not do that on its own.
A body that simply stops between two events, without an error, is the hard case: HTTP/1.1 ends a body without Content-Length or chunked encoding by closing the connection (RFC 9112, section 6.3), so such a cut looks like a normal end. Only the terminal event tells a complete stream from a cut one, which is why the provider methods declare it and generation checks its own CompletedEvent.
Why a broken stream is not retried automatically. Opening the stream again would send the request again. For a generation endpoint that is a new, separately billed generation with different output. The library cannot know what your code has already done with the items it received, such as printing them or forwarding them to a user, so it cannot splice the two streams into one without duplicating or losing text.
Why the SSE resume protocol is not used. SSE has its own way to continue a stream. The server marks events with id:, the client reconnects after retry: milliseconds and sends the last id in the Last-Event-ID header, and the server replays what came after it. The library parses id and retry; you can read them on ServerSentEvent items. It does not reconnect, for three reasons:
- It only works if the server keeps the events and replays them. AI providers' generation streams generally don't. Where a provider supports resuming, it does so with its own parameters (OpenAI's background responses, for example, are resumed from an event's
sequence_numberwithstarting_after). That is a provider method, not a transport feature. - Resuming is a policy, not a request. It needs a number of attempts, a delay, the last id and a way to continue the same
Stream, and each of these depends on the server. A single call can already sendLast-Event-IDwithheaders=; deciding when to send it again is the caller's part for now. - Silent reconnection hides failures. A client that reconnects on its own can loop for a long time against a broken server, while the caller sees a stream that is merely slow.
What to do instead. Decide in your code what a broken stream means. For short answers the simplest choice is to start over and drop the partial text:
from aiollm import (
APIConnectionError,
APIStreamInterruptedError,
APITimeoutError,
)
async def generate_text(prompt: str, attempts: int = 3) -> str:
for attempt in range(1, attempts + 1):
pieces: list[str] = []
try:
async with session.stream(Generate(prompt=prompt)) as stream:
async for piece in stream:
pieces.append(piece)
except (
APIConnectionError, # opening failed
APITimeoutError,
APIStreamInterruptedError, # broke after it was opened
):
if attempt == attempts:
raise
continue # drop the partial text, ask again
return "".join(pieces)
raise ValueError("attempts must be at least 1")
For a provider that can resume, keep the position from the last event you received (its id, or a sequence number in its data) and call the provider's resume method with it. For a server that replays SSE events, open the stream again with the last id: session.stream(method, headers={"Last-Event-ID": last_id}).
An opt-in reconnection with Last-Event-ID for servers that support it is on the roadmap: limited attempts, the retry delay, and the same Stream object continuing with the replayed events.