Shared generation¶
client.generation is the provider-neutral way to generate text: a GenerationResource that takes the fields of a shared GenerationRequest as keyword arguments, as the providers' SDKs do (or a ready GenerationRequest, not both), and returns shared types, with the provider's own response in raw. OpenAIClient and AnthropicClient both have it, and both are an IGenerationClient, so the code below works with either; with an AnthropicClient, raw is a Message. All of it is imported from aiollm.
from typing import Any
from aiollm import (
CompletedEvent,
GenerationRequest,
GenerationResponse,
IGenerationClient,
Turn,
)
async def generate(client: IGenerationClient) -> GenerationResponse[Any]:
return await client.generation.create(
model="gpt-6-astra",
input="Hi",
instructions="Be brief",
max_output_tokens=64,
)
async def generate_streamed(
client: IGenerationClient,
) -> tuple[str, CompletedEvent[Any]]:
text = ""
async with client.generation.stream(
model="gpt-6-astra", input="Hi"
) as events:
async for delta in events.text():
text += delta
completed = await events.final()
return text, completed
async def chat(client: IGenerationClient) -> GenerationResponse[Any]:
first = await client.generation.create(model="gpt-6-astra", input="Hi")
request = GenerationRequest(
model="gpt-6-astra",
input=[Turn.user("Hi"), first.to_turn(), Turn.user("Tell me more")],
)
return await client.generation.create(request)
What each shared field becomes for each provider, what is lost and when a request raises is listed in docs/unification.md.
A user turn can hold images and PDF documents next to its text: its content is then a list of parts, TextPart, ImagePart and DocumentPart, all from aiollm.
from pathlib import Path
from aiollm import DocumentPart, ImagePart, TextPart
async def describe(
client: IGenerationClient, image: Path, report: Path
) -> GenerationResponse[Any]:
question = Turn.user(
[
TextPart(text="Does the chart match the report?"),
await ImagePart.load(image),
await DocumentPart.load(report),
ImagePart(url="https://example.com/logo.png"),
]
)
return await client.generation.create(
model="gpt-6-astra", input=[question]
)
- Bytes or a URL. An
ImagePartor aDocumentPartholds the file's bytes, sent inline in base64, or a URL the provider downloads it from, exactly one of them.ImagePart(data=..., media_type="image/png")takes bytes already in memory; JPEG, PNG, GIF and WebP are the formats both providers take. A document is a PDF;DocumentPart(data=..., filename=...)names it: OpenAI'sfilename(document.pdfotherwise) and Anthropic'stitle. - Reading a file.
await ImagePart.load(path)andawait DocumentPart.load(path)read the file in a worker thread, so the event loop goes on; an image's format comes from the file's extension (media_type=gives it otherwise, and an unknown extension raisesValueError), a document's name from the file. The bytes stay in the part, so a request sent again, by a retry or in the next turn of a conversation, does not read the file again. The format is taken on trust: the extension is not checked against the file's content (a JPEG namedphoto.pngis sent asimage/png), andDocumentPart.loadreads any file as a PDF; a wrong one is refused by the provider. - Only the user's turns. Parts go in
userturns; anassistantturn is text, and parts in it raiseValidationError. The providers limit the size of an image, a document and a request, and answer 400 past them; the parts are not checked against them. A file is read whole, and while a request is built its bytes, their base64 (a third larger) and the JSON body are in memory together. The default encoder, orjson, briefly holds a buffer far larger than the body it returns for a long string, so the peak while a request is encoded is about 15 times the file's size (measured with orjson 3.13 on files of 10 MB); with the standard library's encoder (OpenAIClient(token, json_dumps=json.dumps), any client takes it, see JSON codec) it is about 5 times. Check a file's size before loading it where that matters. What each part becomes is listed under Content parts.
Provider options are typed by the concrete client:
from aiollm import GenerationResponse
from aiollm_openai import OpenAIClient
from aiollm_openai.types import Response
async def generate_unstored(
client: OpenAIClient,
) -> GenerationResponse[Response]:
return await client.generation.create(
model="gpt-6-astra",
input="Hi",
provider_options={"store": False, "temperature": 0.2},
)
- Keyword arguments or a request.
create(model=..., input=...)andstream(...)take exactly the fields ofGenerationRequest; a misspelled field is a type error and, at run time, a pydanticValidationError. Passing aGenerationRequestand fields together raisesTypeError. - Effort.
effort=(anEffortfromaiollm:LOW,MEDIUM,HIGH,XHIGH,MAX) sets how much work the model puts into the response, OpenAI'sreasoning.effortand Anthropic'soutput_config.effort. Left out, it is not sent and the model's default applies; a model that does not take a level answers 400. OpenAI'snoneandminimalare not shared, since Anthropic has nothing like them. - Structured output.
output_schema=takes a JSON Schema (adict; theJSONSchematype is inaiollm.types.requests), and the responsetextis then JSON that follows it: OpenAI'stext.formatwithstrict: true, Anthropic'soutput_config.format. OpenAI requires a name for the format:OpenAIGenerationAdapter(schema_name=...)sets it,"output"by default. The schema is sent as it is; each provider supports its own subset of JSON Schema and answers 400 to the rest. A reply cut off bymax_output_tokensis unfinished JSON, withincomplete_reasonMAX_TOKENS, so check it before parsing. - Reasoning.
response.reasoningandCompletedEvent.reasoninghold a summary of the model's reasoning, and a stream yields it piece by piece asReasoningDeltaEvents (events.text()skips them). A provider sends a summary only when asked, throughprovider_options: OpenAI's{"reasoning": {"summary": "auto"}}, which the adapter merges with the sharedeffortinto onereasoning(settingeffortthere too raisesRequestValidationError), and Anthropic's{"thinking": {"type": "adaptive", "display": "summarized"}}. Without one,reasoningisNonefor both. OpenAI's raw reasoning text and encrypted reasoning, and Anthropic's signatures, stay inraw. - End user.
end_user_id=is an opaque id of the end user the request is made for, such as a hash of an account id, never personal data: OpenAI'ssafety_identifier(up to 128 characters) and Anthropic'smetadata.user_id(up to 512); both use it to detect abuse. OpenAI's free-formmetadatais a provider option. - Tools.
tools=,tool_choice=andparallel_tool_calls=let the model call functions of your code,response.tool_callsholds its calls, andresponse.to_turn()andToolResults carry the loop on; see Tools. - Built-in tools.
tools=also takes tools the provider runs itself,WebSearch(),CodeExecution()and a provider's ownProviderTool(...); the calls are inresponse.server_tool_calls, the sources inresponse.citations, and a long turn the provider paused hasincomplete_reasonPAUSED; see Built-in tools. - Provider options. A field that only one provider has, which the shared request does not hold, goes in
provider_options=ofcreate(...)orstream(...), for that call only. The options are aTypedDictof the provider (OpenAIGenerationOptions:metadata,reasoning,store,previous_response_id,temperature,top_p,include,max_tool_calls;AnthropicGenerationOptions:stop_sequences,thinking), so a wrong key or value is a type error; at run time an unknown key, a shared field among them, or an invalid value raisesRequestValidationErrorbefore anything is sent. A key left out is not sent. Code typed asIGenerationClientcannot pass them, since it works with any provider: itscreatehas no such parameter (example below). - Call options.
create(...)andstream(...)also takeheaders=,timeout=andretry_policy=, assend_requestdoes (see Per-request headers and timeout), and pass them to the session unchanged, so they work the same for every provider. They are options of the call, not fields of the request: no field ofGenerationRequestis named like one of them. - Stream helpers.
stream(...)yields aGenerationStream: iterate it for every shared event, or useevents.text()for the text deltas only andawait events.final()for theCompletedEvent, which carries the response'sid,model, wholetext,usage,tool_callsand why it stopped. Tool calls stream asToolCallStartEvent,ToolCallDeltaEventandToolCallDoneEvent(Tools).final()reads the rest of the stream, and aftertext()or a loop it returns theCompletedEventalready read. When a read failed (a provider's error event, a cut connection), every later read,text()andfinal()raise that same error again, never a misleading "ended without a CompletedEvent". - Any provider.
IGenerationClient[RawT = Any, RawItemT = Any](fromaiollm) is a client withgeneration; every provider client matches it, so code typed with it takes any of them. - A conversation is a list of items.
inputis a text, taken as oneuserturn, or a list ofTurns (Turn.user(...), with a text or parts,Turn.assistant(...), orresponse.to_turn()for an earlier answer), withToolCallTurns andToolResults in a tool loop (see Tools); the list must not be empty, and it ends with auserturn or a tool result (or, to continue a paused turn of built-in tools, theassistantturnto_turn()made of it; see Built-in tools). System prompts go ininstructions, not in turns. - Optional fields are left out until given.
instructions,max_output_tokens,effort,output_schema,end_user_id,tools,tool_choiceandparallel_tool_callsdefault toMISSING, and the adapters pass them on as they are, so a provider gets only what was given. They take noNone, which would meannull: Anthropic's schema has nonullfor them, socreate(..., instructions=None)is a type error and aValidationError. A field not given reads asMISSING, which is truthy: test it withrequest.instructions is MISSING, notif request.instructions. Setting a field later (request.instructions = None) is validated too;request.model_copy(update=...)is not, as everywhere in Pydantic, so a value it lets in reaches the adapter: aNoneis sent asnullto OpenAI, whose schema takes it, and raisesRequestValidationErrorfor Anthropic, whose schema does not. - A request a provider cannot take raises
RequestValidationError(fromaiollm, aValueError) before anything is sent, with theproviderand the sharedfieldat fault; a pydantic error behind it is its__cause__. Errors ofGenerationRequestitself stay pydanticValidationErrors. - Why it stopped.
incomplete_reasonis anIncompleteReason(MAX_TOKENS,REFUSAL,PAUSEDfor a turn of built-in tools the provider paused, orOTHER;Noneif the generation finished), andraw_incomplete_reasonkeeps the provider's own reason. A refusal is an answer, not an error, as in both SDKs: it setsincomplete_reasontoREFUSALandrefusalto the provider's explanation. - OpenAI mapping.
OpenAIGenerationAdapter(aiollm_openai.adapters) buildsCreateResponseorStreamResponsefrom the request, turns asEasyInputMessageitems, so OpenAI's limits, such asmax_output_tokensof at least 16, are checked before sending. Parameters the shared request lacks reach it throughprovider_options(OpenAIGenerationOptions), or stay on the methods above. - OpenAI stopped early. An
incompleteresponse, cut off bymax_output_tokensfor example, keeps itstextand getsincomplete_reason(MAX_TOKENSformax_output_tokens,OTHERfor any other reason) with OpenAI's reason inraw_incomplete_reason; in a stream,response.incompleteis aCompletedEventwith them. Arefusalcontent part makes the response a refusal, with the part's text inrefusal. - OpenAI failures are raised. A
failedresponse raises theAPIStatusErrordescribed under the Responses API, before the adapter sees it, increateand instreamalike. Any other buffered status that is notcompletedorincomplete(cancelled,in_progress, ...) raisesAPIResponseError, since there is no generated text to return. - Anthropic mapping.
AnthropicGenerationAdapter(aiollm_anthropic.adapters) sends a textinputas oneuserturn, a list of turns asmessages, andinstructionsassystem. Anthropic requiresmax_tokens, so a request withoutmax_output_tokensgets the client'sdefault_max_tokens(AnthropicClient(..., default_max_tokens=16000); thinking, always on for the 5.x models, counts toward it, so a lower value can end a reply before its text).end_user_idbecomesmetadata.user_id. - Anthropic counts and stops.
input_tokensis the sum of the input outside the cache and the input read from and written to it (cache_read_tokens,cache_write_tokens), andreasoning_tokensare the thinking tokens. Astop_reasonother thanend_turn,stop_sequenceortool_use(the model waits for tool results) stops the generation early, with the text kept:max_tokensisMAX_TOKENS,refusalisREFUSAL(withstop_details.explanation, else itscategory, inrefusal) and any other isOTHER, with thestop_reasoninraw_incomplete_reason. In a stream,CompletedEventcomes onmessage_stop, with the input counts ofmessage_startand the last cumulative counts ofmessage_delta; a message without astop_reasonraisesAPIResponseError. Anthropic reports failures aserrorevents, which are raised asAPIStatusError. - The end is checked for every provider.
streamyields exactly oneCompletedEvent, as the last event: a stream that ends without it raisesAPIStreamInterruptedError(the provider method's terminal check usually raises it first), and an event after it raisesAPIResponseError. When the provider's terminal proved the stream complete and the adapter still gave noCompletedEvent, the adapter lost it: that raisesRuntimeErrornaming the adapter, not an interruption. A failure is always an exception, never an event.