Skip to content

aiollm.testing

testing

A test kit for code built on aiollm: doubles, fixtures, a local server.

The doubles need no server: FakeTransport replays queued responses through the real BaseSession, LocalAPI builds URLs and make_method declares a method class on the fly. The local server (Recorder in aiollm.testing.aiohttp) answers only what a test scripted with Reply; the fixtures that run it are in the pytest plugin aiollm.testing.plugin. Install the testing extra for them; importing aiollm loads none of this.

Modules:

Name Description
aiohttp

The local test server, an aiohttp application that follows a script.

contract

Shared scenarios that every provider package runs on its client.

doubles

Test doubles that need no server: an API, a transport and methods.

echo

The echo provider: a provider package made for tests.

extension

The extension API: what a provider package may take from the core.

plugin

pytest fixtures for testing code built on aiollm.

recording

What the local test server receives and what it answers.

wire

Checks of a provider's types against the snapshot of its upstream schemas.

Type Aliases:

Name Description
Chunk

A streamed chunk, or an event the server waits on before going on.

Outcome

A queued transport result: returned as is, or raised if an exception.

Classes:

Name Description
FakeStreamResponse

IStreamResponse that yields a fixed list of chunks.

FakeTransport

BaseSession whose transport replays queued outcomes.

LocalAPI

IAPI that joins a base URL, a version and a method path.

RecordedPart

One multipart part as the server received it.

RecordedRequest

A request as the server received it.

Reply

What the server answers to one request.

Functions:

Name Description
make_method

Declare a method class with type(...).

Attributes:

Name Type Description
NOW Final

The current time while the frozen_now fixture is active.

TEST_TOKEN Final

The token the test sessions authenticate with.

NOW module-attribute

NOW: Final = datetime.datetime(
    2026, 1, 2, 3, 4, 5, tzinfo=datetime.UTC
)

The current time while the frozen_now fixture is active.

TEST_TOKEN module-attribute

TEST_TOKEN: Final = 'test-token'

The token the test sessions authenticate with.

Outcome

Outcome = R | BaseException

A queued transport result: returned as is, or raised if an exception.

Chunk

Chunk = bytes | Event

A streamed chunk, or an event the server waits on before going on.

FakeStreamResponse

FakeStreamResponse(
    chunks: Iterable[bytes] = (),
    *,
    status: int = 200,
    headers: Mapping[str, str] | None = None,
    charset: str | None = None,
    fail_at: int | None = None,
    error: BaseException | None = None,
)

IStreamResponse that yields a fixed list of chunks.

Parameters:

Name Type Description Default
chunks Iterable[bytes]

The body, chunk by chunk.

()
status int

HTTP status code.

200
headers Mapping[str, str] | None

Response headers.

None
charset str | None

Charset from the Content-Type header.

None
fail_at int | None

Index of the chunk at which error is raised instead of yielding it; None never fails.

None
error BaseException | None

The exception raised at fail_at.

None

Attributes:

Name Type Description
chunks

The body, chunk by chunk.

fail_at

Index of the chunk that fails, if any.

error

The exception raised there.

aclose_calls

How many times aclose() was called.

read_calls

How many times read() was called.

Raises:

Type Description
ValueError

If only one of fail_at and error is given.

Keep the body and the failure to replay.

Methods:

Name Description
iter_chunks

Yield the chunks, raising error at fail_at.

read

Read the whole body.

aclose

Count a close; closing again is allowed.

status property

status: int

HTTP status code.

headers property

headers: Mapping[str, str]

Response headers, case-insensitive.

charset property

charset: str | None

Charset from the Content-Type header.

closed property

closed: bool

Whether aclose() has been called at least once.

iter_chunks async

iter_chunks() -> AsyncIterator[bytes]

Yield the chunks, raising error at fail_at.

Yields:

Type Description
AsyncIterator[bytes]

Each chunk of the body.

Raises:

Type Description
BaseException

error, at the chunk fail_at.

read async

read() -> bytes

Read the whole body.

Returns:

Type Description
bytes

Every chunk, joined.

aclose async

aclose() -> None

Count a close; closing again is allowed.

FakeTransport

FakeTransport(
    responses: Iterable[Outcome[BufferedResponse]] = (),
    stream_responses: Iterable[
        Outcome[FakeStreamResponse]
    ] = (),
    *,
    api: IAPI | None = None,
    auth: IAuth | None = None,
    error_resolver: IErrorResolver | None = None,
    **kwargs: Any,
)

Bases: BaseSession

BaseSession whose transport replays queued outcomes.

_execute pops from responses and _open_stream from stream_responses; an exception in the queue is raised instead of returned, and an empty queue raises AssertionError. Every call is recorded, so tests can assert exact attempt counts and which stream responses were closed, and the headers and timeout of each attempt go to call_options.

Parameters:

Name Type Description Default
responses Iterable[Outcome[BufferedResponse]]

Outcomes of _execute, in order.

()
stream_responses Iterable[Outcome[FakeStreamResponse]]

Outcomes of _open_stream, in order.

()
api IAPI | None

URL builder; never used by the fake transport itself.

None
auth IAuth | None

Authorization header source; a bearer TEST_TOKEN if not given.

None
error_resolver IErrorResolver | None

Turns error responses into exceptions; the base resolver if not given.

None
**kwargs Any

Forwarded to BaseSession (headers, JSON codecs, retry policy).

{}

Attributes:

Name Type Description
responses deque[Outcome[BufferedResponse]]

Outcomes _execute has not replayed yet.

stream_responses deque[Outcome[FakeStreamResponse]]

Outcomes _open_stream has not replayed yet.

execute_calls list[IAPIMethod[Any]]

Every method _execute was called with.

stream_calls list[IStreamMethod[Any]]

Every method _open_stream was called with.

opened list[FakeStreamResponse]

Every stream response _open_stream returned.

call_options list[tuple[HeadersType | None, Timeout | None]]

The headers and timeout of each attempt.

close_calls

How many times close() was called.

Build the session and queue the outcomes.

Methods:

Name Description
close

Count a close; nothing is open.

attempts property

attempts: int

Number of _execute and _open_stream calls.

closed_responses property

closed_responses: int

Number of opened stream responses that were closed.

close async

close() -> None

Count a close; nothing is open.

LocalAPI dataclass

LocalAPI(base_url: URL, version: str = 'v1')

IAPI that joins a base URL, a version and a method path.

Attributes:

Name Type Description
base_url URL

The server's URL, such as the local test server's.

version str

The path segment before every method path.

Methods:

Name Description
build_url

Build the URL of a request.

build_url

build_url(method: str, query: Query | None = None) -> str

Build the URL of a request.

Parameters:

Name Type Description Default
method str

The method's path, already percent-encoded.

required
query Query | None

The query parameters, if any.

None

Returns:

Type Description
str

The full URL.

RecordedPart dataclass

RecordedPart(
    name: str | None,
    filename: str | None,
    content_type: str | None,
    data: bytes,
)

One multipart part as the server received it.

Attributes:

Name Type Description
name str | None

The part's name in Content-Disposition.

filename str | None

The part's filename, if it has one.

content_type str | None

The part's Content-Type, if it has one.

data bytes

The part's body.

RecordedRequest dataclass

RecordedRequest(
    method: str,
    raw_path: str,
    query_string: str,
    query: MultiDict[str],
    headers: CIMultiDictProxy[str],
    body: bytes,
    parts: list[RecordedPart],
)

A request as the server received it.

Attributes:

Name Type Description
method str

The HTTP method.

raw_path str

The path exactly as sent, still percent-encoded and without the query string.

query_string str

The query string as sent.

query MultiDict[str]

The query parameters, decoded.

headers CIMultiDictProxy[str]

The request headers.

body bytes

The body; empty for a multipart body, which is split into parts.

parts list[RecordedPart]

The parts of a multipart body.

Reply dataclass

Reply(
    status: int = 200,
    body: bytes = b"",
    json: Any = None,
    headers: Mapping[str, str] = dict[str, str](),
    content_type: str | None = None,
    chunks: Iterable[Chunk] | None = None,
    delay: float = 0.0,
    chunk_delay: float = 0.0,
    content_length: int | None = None,
    abort: bool = False,
)

What the server answers to one request.

Attributes:

Name Type Description
status int

HTTP status code.

body bytes

Whole body, sent with Content-Length.

json Any

Body serialized as JSON; overrides body.

headers Mapping[str, str]

Extra response headers.

content_type str | None

Content-Type, charset included.

chunks Iterable[Chunk] | None

Streamed body (chunked encoding); an asyncio.Event item makes the server wait for it. May be infinite.

delay float

Seconds to wait before answering.

chunk_delay float

Seconds to wait before each chunk.

content_length int | None

Declared Content-Length for chunks; with abort it makes the body end before it is complete.

abort bool

Drop the connection after the chunks are sent.

make_method

make_method(
    base: type[M],
    *,
    path: str | None = "/test",
    fields: Mapping[str, Any] | None = None,
    name: str = "Method",
    http_method: str = "POST",
    **class_vars: Any,
) -> type[M]

Declare a method class with type(...).

Meant for parametrized class-definition checks; wrap the call in pytest.raises when the definition must fail. Type checkers only see base, so create instances with dynamic fields through model_validate({...}).

Parameters:

Name Type Description Default
base type[M]

Base class, e.g. BaseAPIMethod[str].

required
path str | None

__request_path__; None inherits it from base.

'/test'
fields Mapping[str, Any] | None

Field annotations; a (type, default) tuple also sets the default.

None
name str

Class name, shown in error messages.

'Method'
http_method str

__request_method__.

'POST'
**class_vars Any

Other class attributes, e.g. __request_query__={"q"}.

{}

Returns:

Type Description
type[M]

The new method class.

Example

GetItem = make_method( ... BaseAPIMethod[str], ... path="/items/{item_id}", ... fields={"item_id": str}, ... http_method="GET", ... ) GetItem.model_validate({"item_id": "a/b"}).request_path '/items/a%2Fb'

aiohttp

The local test server, an aiohttp application that follows a script.

Transport-specific, like every aiohttp.py module of the library, so aiollm.testing imports without it.

Classes:

Name Description
Recorder

aiohttp app that records requests and answers from a script.

Recorder

Recorder()

aiohttp app that records requests and answers from a script.

Replies are taken from replies in order; once the script is exhausted, default answers, if a test set one. Without it, the request is unscripted: it gets a 500 and the recorder fixture fails the test, so a retry or a second request nobody planned for cannot pass silently. Every path and HTTP method is routed here.

Attributes:

Name Type Description
requests list[RecordedRequest]

Every request received, in order.

replies deque[Reply]

The replies not given yet.

default Reply | None

The reply once replies is empty, if any.

unscripted list[RecordedRequest]

The requests no reply was scripted for.

app

The aiohttp application to serve.

Start with an empty script and no default reply.

Methods:

Name Description
reply

Queue replies for the next requests.

last property

The most recent request.

Raises:

Type Description
AssertionError

If no request was received.

reply

reply(*replies: Reply) -> None

Queue replies for the next requests.

Parameters:

Name Type Description Default
*replies Reply

The replies, in the order of the requests.

()

plugin

pytest fixtures for testing code built on aiollm.

Load the plugin from a conftest.py with pytest_plugins = ["aiollm.testing.plugin"]; it is not registered as an entry point, since its autouse fixtures change how the library behaves in every test. It needs the testing extra (pytest, pytest-asyncio and pytest-aiohttp).

Autouse: no_retry_wait (retries do not wait), no_provider_keys (the variables provider_key_envs names are cleared) and no_close_delay (AiohttpSession.close() does not pause). On request: frozen_now, fake_transport, and the local server recorder, server, api and session; recorder fails the test at teardown if a request came that no reply was scripted for.

Functions:

Name Description
no_retry_wait

Make retries instant.

provider_key_envs

Environment variables no_provider_keys clears.

no_provider_keys

Hide API keys of the environment, so clients see only test tokens.

no_close_delay

Drop the 0.25 s pause of AiohttpSession.close().

frozen_now

Fix the current time at NOW for the library's clock reads.

fake_transport

A FakeTransport with empty queues; fill them in the test.

recorder

A fresh request recorder; api serves it.

server

The recorder running on a local port.

api

IAPI pointing at the local server.

session

AiohttpSession against the local server, closed on teardown.

no_retry_wait

no_retry_wait() -> Iterator[MagicMock]

Make retries instant.

Replaces calculate_wait where the retry loop looks it up, so it waits 0 seconds and the policy's backoff is not called. Tenacity reuses one mutable state object, so the mock's call_args only show the final state; the error each wait was computed for is kept in mock.outcomes instead. A test that needs the real wait sets the mock's side_effect to calculate_wait imported from aiollm.session.retry.runner at the top of its module, before the patch.

Yields:

Type Description
MagicMock

The mock, with outcomes.

provider_key_envs

provider_key_envs() -> tuple[str, ...]

Environment variables no_provider_keys clears.

Override it in a conftest.py with the variables a provider client reads its key from, such as ("EXAMPLE_API_KEY",).

Returns:

Type Description
tuple[str, ...]

No variable.

no_provider_keys

no_provider_keys(
    monkeypatch: MonkeyPatch,
    provider_key_envs: tuple[str, ...],
) -> None

Hide API keys of the environment, so clients see only test tokens.

Parameters:

Name Type Description Default
monkeypatch MonkeyPatch

pytest's monkeypatch.

required
provider_key_envs tuple[str, ...]

The variables to clear.

required

no_close_delay

no_close_delay(monkeypatch: MonkeyPatch) -> None

Drop the 0.25 s pause of AiohttpSession.close().

Only the delay constant is replaced: the module and its asyncio stay real, so everything else in it behaves as in production.

Parameters:

Name Type Description Default
monkeypatch MonkeyPatch

pytest's monkeypatch.

required

frozen_now

frozen_now(monkeypatch: MonkeyPatch) -> datetime

Fix the current time at NOW for the library's clock reads.

Replaces http.headers._now, the only place the library reads the clock (Retry-After dates), and nothing else.

Parameters:

Name Type Description Default
monkeypatch MonkeyPatch

pytest's monkeypatch.

required

Returns:

Type Description
datetime

NOW.

fake_transport

fake_transport() -> FakeTransport

A FakeTransport with empty queues; fill them in the test.

Returns:

Type Description
FakeTransport

The transport.

recorder

recorder() -> Iterator[Recorder]

A fresh request recorder; api serves it.

Yields:

Type Description
Recorder

The recorder.

Raises:

Type Description
AssertionError

At teardown, if a request came that no reply was scripted for.

server async

server(
    aiohttp_server: AiohttpServer, recorder: Recorder
) -> TestServer

The recorder running on a local port.

Parameters:

Name Type Description Default
aiohttp_server AiohttpServer

pytest-aiohttp's server factory.

required
recorder Recorder

The recorder to serve.

required

Returns:

Type Description
TestServer

The running server.

api

api(server: TestServer) -> LocalAPI

IAPI pointing at the local server.

Parameters:

Name Type Description Default
server TestServer

The running server.

required

Returns:

Type Description
LocalAPI

The API.

session async

AiohttpSession against the local server, closed on teardown.

Parameters:

Name Type Description Default
api LocalAPI

The local server's API.

required

Yields:

Type Description
AsyncIterator[AiohttpSession]

The session.

generation

Scenarios every provider's client.generation passes.

A provider's tests subclass GenerationContract with a GenerationCase of its wire format; every scenario then runs against the provider's client on the local server (aiollm.testing.plugin fixtures), with the provider's own payloads. Only the wire format differs per provider: the shared results and errors must not.

Example

from aiollm.testing.contract import ( ... GenerationCase, ... GenerationContract, ... ) EXAMPLE = GenerationCase(provider=PROVIDER, client=ExampleClient, ...) class TestExampleGeneration(GenerationContract): ... case = EXAMPLE

Classes:

Name Description
BuiltinCase

The built-in tool scenarios of one provider.

GenerationCase

One provider in the generation scenarios.

GenerationContract

The generation scenarios, for a subclass to run on its provider.

Functions:

Name Description
with_tools

Build the request of most scenarios with tools.

frame

Write one SSE frame with event: and data:.

events

Build a streamed reply of the given events.

concurrently

Run one call per options at once on the client's one resource.

text_of

Take the text deltas among shared events.

endless

Build a stream of text deltas that never ends.

opener

Open the raw provider stream or the shared generation stream.

Attributes:

Name Type Description
REQUEST Final

The request of most scenarios.

SCHEMA Final[JSONSchema]

Parameters of the tools of the scenarios.

CALLS Final[list[tuple[str, str, dict[str, Any]]]]

Two calls the model makes in one response: id, tool, arguments.

TOOLS Final

A strict tool with a description, and a bare one.

PNG Final

The bytes of the image part of PARTS.

PDF Final

The bytes of the document parts of PARTS.

PARTS Final[list[ContentPart]]

A user turn of every kind of part and source.

BUILTIN_TOOLS Final

The built-in tools of the scenarios: a search with both its options,

SEARCH_ID Final

The id of the web search call the model makes in the scenarios.

SEARCH_QUERIES Final

What that search looks for.

CITATIONS Final

The citations of the text the search leads to: kind, url, title.

CODE_ID Final

The id of the code execution call the model makes in the scenarios.

PRODUCED_ID Final

The id of the file that code execution makes.

PRODUCED_NAME Final

The name of that file.

CODE_FILE_ID Final

The id of the stored file the scenarios put into the sandbox.

MODES Final

Both streams: the provider's own method and the shared one.

REQUEST module-attribute

REQUEST: Final = GenerationRequest(
    model="model-1",
    input="Hi",
    instructions="Be brief",
    max_output_tokens=64,
)

The request of most scenarios.

SCHEMA module-attribute

SCHEMA: Final[JSONSchema] = {
    "type": "object",
    "properties": {"city": {"type": "string"}},
    "required": ["city"],
    "additionalProperties": False,
}

Parameters of the tools of the scenarios.

CALLS module-attribute

CALLS: Final[list[tuple[str, str, dict[str, Any]]]] = [
    ("call_1", "get_weather", {"city": "Paris"}),
    ("call_2", "get_time", {}),
]

Two calls the model makes in one response: id, tool, arguments.

TOOLS module-attribute

TOOLS: Final = [
    Tool(
        name="get_weather",
        description="Current weather",
        parameters=SCHEMA,
        strict=True,
    ),
    Tool(
        name="get_time",
        parameters={"type": "object", "properties": {}},
    ),
]

A strict tool with a description, and a bare one.

PNG module-attribute

PNG: Final = b'\x89PNG\r\n'

The bytes of the image part of PARTS.

PDF module-attribute

PDF: Final = b'%PDF-1.7'

The bytes of the document parts of PARTS.

PARTS module-attribute

PARTS: Final[list[ContentPart]] = [
    TextPart(text="What is in these?"),
    ImagePart(data=PNG, media_type="image/png"),
    ImagePart(url="https://example.com/cat.jpg"),
    DocumentPart(data=PDF, filename="report.pdf"),
    DocumentPart(data=PDF),
    DocumentPart(url="https://example.com/report.pdf"),
]

A user turn of every kind of part and source.

BUILTIN_TOOLS module-attribute

BUILTIN_TOOLS: Final = [
    WebSearch(
        allowed_domains=["example.com"],
        user_location=UserLocation(
            country="FR", city="Paris"
        ),
    ),
    CodeExecution(),
]

The built-in tools of the scenarios: a search with both its options, and a code execution.

SEARCH_ID module-attribute

SEARCH_ID: Final = 'search_1'

The id of the web search call the model makes in the scenarios.

SEARCH_QUERIES module-attribute

SEARCH_QUERIES: Final = ['weather in Paris']

What that search looks for.

CITATIONS module-attribute

CITATIONS: Final = [
    (
        CitationKind.URL,
        "https://example.com/paris",
        "Paris weather",
    )
]

The citations of the text the search leads to: kind, url, title.

CODE_ID module-attribute

CODE_ID: Final = 'code_1'

The id of the code execution call the model makes in the scenarios.

PRODUCED_ID module-attribute

PRODUCED_ID: Final = 'file_made_1'

The id of the file that code execution makes.

PRODUCED_NAME module-attribute

PRODUCED_NAME: Final = 'chart.png'

The name of that file.

CODE_FILE_ID module-attribute

CODE_FILE_ID: Final = 'file_code_1'

The id of the stored file the scenarios put into the sandbox.

MODES module-attribute

MODES: Final = pytest.mark.parametrize(
    "mode", ["raw", "generation"]
)

Both streams: the provider's own method and the shared one.

BuiltinCase dataclass

BuiltinCase(
    tools: dict[str, Any],
    mixed: dict[str, Any],
    searched: dict[str, Any],
    searched_stream: list[dict[str, Any]],
    failed: dict[str, Any],
    coded: dict[str, Any],
    coded_stream: list[dict[str, Any]],
    produced: dict[str, Any],
    code_files: dict[str, Any],
    paused: dict[str, Any] | None = None,
    continued: list[Any] | None = None,
)

The built-in tool scenarios of one provider.

Every dict is in the provider's wire format. A provider that runs built-in tools gives its GenerationCase one of these; a case without it skips the scenarios.

Attributes:

Name Type Description
tools dict[str, Any]

What BUILTIN_TOOLS become in the body: the keys a request with them (and ToolChoice.AUTO) sets.

mixed dict[str, Any]

What a function tool, TOOLS[1], followed by WebSearch() become: the function tool goes first.

searched dict[str, Any]

A finished response of one web search, call SEARCH_ID for SEARCH_QUERIES, with the text of GenerationCase.text and the CITATIONS.

searched_stream list[dict[str, Any]]

A finished stream of the same.

failed dict[str, Any]

A finished response whose web search failed.

coded dict[str, Any]

A finished response of one code execution, call CODE_ID, that made file PRODUCED_ID (named PRODUCED_NAME, if the provider names it), with text.

coded_stream list[dict[str, Any]]

A finished stream of the same.

produced dict[str, Any]

The fields of that ProducedFile the provider fills besides provider and id: the filename, the container_id, the source_call_id (CODE_ID), as far as it tells them.

code_files dict[str, Any]

What CodeExecution(files=[...]) of stored file CODE_FILE_ID becomes in the body of REQUEST: the keys it sets.

paused dict[str, Any] | None

A response that paused its turn in the middle of a search, without text; its native output goes back. Left out by a provider whose API never pauses a turn, which skips that scenario.

continued list[Any] | None

The body's conversation after its first turn, when paused comes back with the user turn before it; given with paused.

GenerationCase dataclass

GenerationCase(
    provider: Provider,
    client: Callable[..., Any],
    factory: SessionFactory[Any, Any, ...],
    path: str,
    response: dict[str, Any],
    text: str,
    usage: TokenUsage,
    truncated: dict[str, Any],
    refused: dict[str, Any],
    refusal: str,
    stream: list[dict[str, Any]],
    deltas: list[str],
    stream_usage: TokenUsage,
    overloaded: dict[str, Any],
    quota: dict[str, Any],
    bad_request: dict[str, Any],
    raw_stream: Callable[[], BaseStreamMethod[Any]],
    options: dict[str, Any],
    other_options: dict[str, Any],
    bad_option: dict[str, Any],
    effort: dict[str, Any],
    output_schema: dict[str, Any],
    tools: dict[str, Any],
    forced: dict[str, Any],
    only_parallel: dict[str, Any],
    called: dict[str, Any],
    called_stream: list[dict[str, Any]],
    cut_off: dict[str, Any],
    input_field: str,
    continued: list[Any],
    continued_shared: list[Any],
    continued_streamed: list[Any],
    parts: dict[str, Any],
    after_results: list[Any],
    builtin: BuiltinCase | None = None,
)

One provider in the generation scenarios.

Attributes:

Name Type Description
provider Provider

The provider.

client Callable[..., Any]

Builds the client from a token, or from session=.

factory SessionFactory[Any, Any, ...]

The session factory of the client, sent to the local server by replacing its api.

path str

Request path of the endpoint, such as /v1/responses.

response dict[str, Any]

A finished buffered response.

text str

Its text.

usage TokenUsage

Its shared token counts.

truncated dict[str, Any]

A response cut off by the token limit, text kept.

refused dict[str, Any]

A response in which the model refused.

refusal str

The refusal's text in it.

stream list[dict[str, Any]]

Events of a finished stream, its last event terminal; its first event whose type has delta in it is a text delta.

deltas list[str]

Text deltas of that stream.

stream_usage TokenUsage

Shared token counts of that stream.

overloaded dict[str, Any]

An overloaded error, as the stream sends it.

quota dict[str, Any]

An exhausted quota, as the stream sends it.

bad_request dict[str, Any]

Body of a 400 response.

raw_stream Callable[[], BaseStreamMethod[Any]]

Builds the provider's own stream method.

options dict[str, Any]

Valid provider options, sent as they are.

other_options dict[str, Any]

Other valid options, for a concurrent call.

bad_option dict[str, Any]

A provider option with a value the method refuses.

effort dict[str, Any]

What effort=Effort.HIGH becomes in the body.

output_schema dict[str, Any]

What an output schema with Effort.HIGH becomes.

tools dict[str, Any]

What TOOLS with ToolChoice.REQUIRED and no parallel calls become.

forced dict[str, Any]

What a ForcedTool becomes.

only_parallel dict[str, Any]

What parallel_tool_calls=True alone becomes.

called dict[str, Any]

A finished response with CALLS, text first.

called_stream list[dict[str, Any]]

A finished stream with CALLS.

cut_off dict[str, Any]

What makes called stop at the token limit.

input_field str

The body field of the conversation.

continued list[Any]

The body's conversation after its first turn, when called comes back with its results.

continued_shared list[Any]

The same from the shared calls only.

continued_streamed list[Any]

The same when called_stream comes back.

parts dict[str, Any]

What a user turn of PARTS becomes.

after_results list[Any]

The body's conversation after its first two items, when a call to get_time (id call_1) has its result noon and a user turn of PARTS[:2] follows.

builtin BuiltinCase | None

The built-in tool scenarios; None for a provider without built-in tools, which skips them.

delta property

delta: dict[str, Any]

The first text delta of the stream.

until_delta property

until_delta: list[dict[str, Any]]

The stream up to its first text delta, included.

GenerationContract

The generation scenarios, for a subclass to run on its provider.

A subclass whose name starts with Test sets case; pytest collects its scenarios. It needs the aiollm.testing.plugin fixtures (recorder, api).

Attributes:

Name Type Description
case GenerationCase

The provider the scenarios run on.

Methods:

Name Description
client

Build the provider's client, sent to the local server.

make_client

Build the provider's client with the given session settings.

test_create_returns_the_text_and_usage

Create returns the text and usage.

test_truncated_response_keeps_text_and_reason

Truncated response keeps text and reason.

test_refusal_is_an_answer

Refusal is an answer.

test_generation_shares_the_client_session

Generation shares the client session.

test_effort_reaches_the_wire

Effort reaches the wire.

test_an_output_schema_reaches_the_wire_as_it_is

An output schema reaches the wire as it is.

test_tools_reach_the_wire

Tools reach the wire.

test_tool_calls_come_back_alike

Tool calls come back alike.

test_tool_calls_stream_as_start_pieces_and_done

Tool calls stream as start pieces and done.

test_a_stream_can_fail_after_its_calls_are_done

A stream can fail after its calls are done.

test_a_turn_of_parts_sends_text_images_and_documents

A turn of parts sends text images and documents.

test_parts_after_tool_results_go_with_them

Parts after tool results go with them.

test_a_streamed_tool_loop_continues_with_the_results

A streamed tool loop continues with the results.

test_a_tool_loop_continues_with_the_results

A tool loop continues with the results.

test_a_turn_of_calls_from_another_provider_sends_its_calls

A turn of calls from another provider sends its calls.

test_calls_and_results_out_of_order_are_refused

Calls and results out of order are refused.

test_a_response_that_stopped_mid_call_is_no_turn

A response that stopped mid call is no turn.

test_a_response_without_calls_has_none

A response without calls has none.

test_tool_choices_reach_the_wire

Tool choices reach the wire.

test_provider_options_reach_the_wire_of_their_call_only

Provider options reach the wire of their call only.

test_concurrent_calls_keep_their_own_provider_options

Concurrent calls keep their own provider options.

test_an_unknown_provider_option_is_refused_before_sending

An unknown provider option is refused before sending.

test_an_invalid_provider_option_is_refused_before_sending

An invalid provider option is refused before sending.

test_conversation_must_end_with_a_user_turn

Conversation must end with a user turn.

test_conversation_continues_with_the_response

Conversation continues with the response.

test_stream_yields_deltas_then_completes

Stream yields deltas then completes.

test_stream_error_event_raises_overloaded

Stream error event raises overloaded.

test_stream_quota_error_is_a_quota_error

Stream quota error is a quota error.

test_stream_cut_before_its_end_raises

Stream cut before its end raises.

test_break_closes_the_stream

Break closes the stream.

test_error_status_raises_before_streaming

Error status raises before streaming.

test_transient_failures_are_retried

Transient failures are retried.

test_call_options_reach_the_session

Call options reach the session.

test_a_call_timeout_limits_the_silence_of_a_stream

A call timeout limits the silence of a stream.

test_connection_drop_keeps_the_text_and_raises

Connection drop keeps the text and raises.

test_silence_keeps_the_text_and_raises

Silence keeps the text and raises.

test_cancelling_the_task_while_iterating_releases_the_connection

Cancelling the task while iterating releases the connection.

test_cancelling_the_task_while_opening_releases_the_connection

Cancelling the task while opening releases the connection.

builtin

Take the case's built-in tool scenarios, or skip.

test_builtin_tools_reach_the_wire

Built-in tools reach the wire.

test_builtin_tools_follow_function_tools

Built-in tools follow function tools.

test_a_tool_of_another_provider_is_refused

A tool of another provider is refused.

test_builtin_tool_calls_come_back_alike

Built-in tool calls and citations come back alike.

test_builtin_tool_calls_stream_as_started_then_done

Built-in tool calls stream as started then done.

test_a_failed_builtin_tool_is_a_call_not_an_error

A failed built-in tool is a call, not an error.

test_a_paused_generation_continues_with_its_native_output

A paused generation continues with its native output.

test_files_a_tool_made_come_back_alike

The files a built-in tool made come back alike.

test_files_for_the_code_reach_the_wire

The files a code execution reads reach the wire.

test_code_execution_refuses_a_file_it_cannot_read

Code execution refuses another provider's or a container's file.

client

client(api: LocalAPI, monkeypatch: MonkeyPatch) -> Any

Build the provider's client, sent to the local server.

Parameters:

Name Type Description Default
api LocalAPI

The local server's API.

required
monkeypatch MonkeyPatch

pytest's monkeypatch.

required

Returns:

Type Description
Any

The client, built from TEST_TOKEN.

make_client

make_client(
    api: LocalAPI, monkeypatch: MonkeyPatch
) -> Callable[..., Any]

Build the provider's client with the given session settings.

Parameters:

Name Type Description Default
api LocalAPI

The local server's API.

required
monkeypatch MonkeyPatch

pytest's monkeypatch.

required

Returns:

Type Description
Callable[..., Any]

A function of the session's keyword arguments that builds

Callable[..., Any]

the client on a session of the factory.

test_create_returns_the_text_and_usage async

test_create_returns_the_text_and_usage(
    client: Any, recorder: Recorder
) -> None

Create returns the text and usage.

test_truncated_response_keeps_text_and_reason async

test_truncated_response_keeps_text_and_reason(
    client: Any, recorder: Recorder
) -> None

Truncated response keeps text and reason.

test_refusal_is_an_answer async

test_refusal_is_an_answer(
    client: Any, recorder: Recorder
) -> None

Refusal is an answer.

test_generation_shares_the_client_session

test_generation_shares_the_client_session(
    client: Any,
) -> None

Generation shares the client session.

test_effort_reaches_the_wire async

test_effort_reaches_the_wire(
    client: Any, recorder: Recorder, mode: str
) -> None

Effort reaches the wire.

test_an_output_schema_reaches_the_wire_as_it_is async

test_an_output_schema_reaches_the_wire_as_it_is(
    client: Any, recorder: Recorder, mode: str
) -> None

An output schema reaches the wire as it is.

test_tools_reach_the_wire async

test_tools_reach_the_wire(
    client: Any, recorder: Recorder, mode: str
) -> None

Tools reach the wire.

test_tool_calls_come_back_alike async

test_tool_calls_come_back_alike(
    client: Any, recorder: Recorder, mode: str
) -> None

Tool calls come back alike.

test_tool_calls_stream_as_start_pieces_and_done async

test_tool_calls_stream_as_start_pieces_and_done(
    client: Any, recorder: Recorder
) -> None

Tool calls stream as start pieces and done.

test_a_stream_can_fail_after_its_calls_are_done async

test_a_stream_can_fail_after_its_calls_are_done(
    client: Any, recorder: Recorder
) -> None

A stream can fail after its calls are done.

test_a_turn_of_parts_sends_text_images_and_documents async

test_a_turn_of_parts_sends_text_images_and_documents(
    client: Any, recorder: Recorder, mode: str
) -> None

A turn of parts sends text images and documents.

test_parts_after_tool_results_go_with_them async

test_parts_after_tool_results_go_with_them(
    client: Any, recorder: Recorder
) -> None

Parts after tool results go with them.

test_a_streamed_tool_loop_continues_with_the_results async

test_a_streamed_tool_loop_continues_with_the_results(
    client: Any, recorder: Recorder
) -> None

A streamed tool loop continues with the results.

test_a_tool_loop_continues_with_the_results async

test_a_tool_loop_continues_with_the_results(
    client: Any, recorder: Recorder
) -> None

A tool loop continues with the results.

test_a_turn_of_calls_from_another_provider_sends_its_calls async

test_a_turn_of_calls_from_another_provider_sends_its_calls(
    client: Any, recorder: Recorder
) -> None

A turn of calls from another provider sends its calls.

test_calls_and_results_out_of_order_are_refused async

test_calls_and_results_out_of_order_are_refused(
    client: Any, items: list[str], match: str
) -> None

Calls and results out of order are refused.

test_a_response_that_stopped_mid_call_is_no_turn async

test_a_response_that_stopped_mid_call_is_no_turn(
    client: Any, recorder: Recorder
) -> None

A response that stopped mid call is no turn.

test_a_response_without_calls_has_none async

test_a_response_without_calls_has_none(
    client: Any, recorder: Recorder
) -> None

A response without calls has none.

test_tool_choices_reach_the_wire async

test_tool_choices_reach_the_wire(
    client: Any,
    recorder: Recorder,
    changes: dict[str, Any],
    expected: str,
) -> None

Tool choices reach the wire.

test_provider_options_reach_the_wire_of_their_call_only async

test_provider_options_reach_the_wire_of_their_call_only(
    client: Any, recorder: Recorder, mode: str
) -> None

Provider options reach the wire of their call only.

test_concurrent_calls_keep_their_own_provider_options async

test_concurrent_calls_keep_their_own_provider_options(
    client: Any, recorder: Recorder, mode: str
) -> None

Concurrent calls keep their own provider options.

test_an_unknown_provider_option_is_refused_before_sending async

test_an_unknown_provider_option_is_refused_before_sending(
    client: Any, recorder: Recorder, mode: str
) -> None

An unknown provider option is refused before sending.

test_an_invalid_provider_option_is_refused_before_sending async

test_an_invalid_provider_option_is_refused_before_sending(
    client: Any, recorder: Recorder, mode: str
) -> None

An invalid provider option is refused before sending.

test_conversation_must_end_with_a_user_turn async

test_conversation_must_end_with_a_user_turn(
    client: Any, recorder: Recorder, mode: str
) -> None

Conversation must end with a user turn.

test_conversation_continues_with_the_response async

test_conversation_continues_with_the_response(
    client: Any, recorder: Recorder
) -> None

Conversation continues with the response.

test_stream_yields_deltas_then_completes async

test_stream_yields_deltas_then_completes(
    client: Any, recorder: Recorder
) -> None

Stream yields deltas then completes.

test_stream_error_event_raises_overloaded async

test_stream_error_event_raises_overloaded(
    client: Any, recorder: Recorder
) -> None

Stream error event raises overloaded.

test_stream_quota_error_is_a_quota_error async

test_stream_quota_error_is_a_quota_error(
    client: Any, recorder: Recorder
) -> None

Stream quota error is a quota error.

test_stream_cut_before_its_end_raises async

test_stream_cut_before_its_end_raises(
    client: Any, recorder: Recorder
) -> None

Stream cut before its end raises.

test_break_closes_the_stream async

test_break_closes_the_stream(
    client: Any, recorder: Recorder
) -> None

Break closes the stream.

test_error_status_raises_before_streaming async

test_error_status_raises_before_streaming(
    client: Any, recorder: Recorder
) -> None

Error status raises before streaming.

test_transient_failures_are_retried async

test_transient_failures_are_retried(
    client: Any, recorder: Recorder
) -> None

Transient failures are retried.

test_call_options_reach_the_session async

test_call_options_reach_the_session(
    client: Any, recorder: Recorder
) -> None

Call options reach the session.

test_a_call_timeout_limits_the_silence_of_a_stream async

test_a_call_timeout_limits_the_silence_of_a_stream(
    client: Any, recorder: Recorder
) -> None

A call timeout limits the silence of a stream.

test_connection_drop_keeps_the_text_and_raises async

test_connection_drop_keeps_the_text_and_raises(
    client: Any, recorder: Recorder
) -> None

Connection drop keeps the text and raises.

test_silence_keeps_the_text_and_raises async

test_silence_keeps_the_text_and_raises(
    make_client: Callable[..., Any], recorder: Recorder
) -> None

Silence keeps the text and raises.

test_cancelling_the_task_while_iterating_releases_the_connection async

test_cancelling_the_task_while_iterating_releases_the_connection(
    make_client: Callable[..., Any],
    recorder: Recorder,
    mode: str,
) -> None

Cancelling the task while iterating releases the connection.

test_cancelling_the_task_while_opening_releases_the_connection async

test_cancelling_the_task_while_opening_releases_the_connection(
    make_client: Callable[..., Any],
    recorder: Recorder,
    mode: str,
) -> None

Cancelling the task while opening releases the connection.

builtin

builtin() -> BuiltinCase

Take the case's built-in tool scenarios, or skip.

Returns:

Type Description
BuiltinCase

The case's builtin.

test_builtin_tools_reach_the_wire async

test_builtin_tools_reach_the_wire(
    client: Any,
    recorder: Recorder,
    builtin: BuiltinCase,
    mode: str,
) -> None

Built-in tools reach the wire.

test_builtin_tools_follow_function_tools async

test_builtin_tools_follow_function_tools(
    client: Any, recorder: Recorder, builtin: BuiltinCase
) -> None

Built-in tools follow function tools.

test_a_tool_of_another_provider_is_refused async

test_a_tool_of_another_provider_is_refused(
    client: Any, recorder: Recorder, builtin: BuiltinCase
) -> None

A tool of another provider is refused.

test_builtin_tool_calls_come_back_alike async

test_builtin_tool_calls_come_back_alike(
    client: Any,
    recorder: Recorder,
    builtin: BuiltinCase,
    mode: str,
) -> None

Built-in tool calls and citations come back alike.

test_builtin_tool_calls_stream_as_started_then_done async

test_builtin_tool_calls_stream_as_started_then_done(
    client: Any, recorder: Recorder, builtin: BuiltinCase
) -> None

Built-in tool calls stream as started then done.

test_a_failed_builtin_tool_is_a_call_not_an_error async

test_a_failed_builtin_tool_is_a_call_not_an_error(
    client: Any, recorder: Recorder, builtin: BuiltinCase
) -> None

A failed built-in tool is a call, not an error.

test_a_paused_generation_continues_with_its_native_output async

test_a_paused_generation_continues_with_its_native_output(
    client: Any, recorder: Recorder, builtin: BuiltinCase
) -> None

A paused generation continues with its native output.

test_files_a_tool_made_come_back_alike async

test_files_a_tool_made_come_back_alike(
    client: Any,
    recorder: Recorder,
    builtin: BuiltinCase,
    mode: str,
) -> None

The files a built-in tool made come back alike.

test_files_for_the_code_reach_the_wire async

test_files_for_the_code_reach_the_wire(
    client: Any, recorder: Recorder, builtin: BuiltinCase
) -> None

The files a code execution reads reach the wire.

test_code_execution_refuses_a_file_it_cannot_read async

test_code_execution_refuses_a_file_it_cannot_read(
    client: Any,
    recorder: Recorder,
    builtin: BuiltinCase,
    kind: str,
) -> None

Code execution refuses another provider's or a container's file.

with_tools

with_tools(
    tools: list[Any], input: Any = None
) -> GenerationRequest

Build the request of most scenarios with tools.

Parameters:

Name Type Description Default
tools list[Any]

The tools: functions, built-in tools, or both.

required
input Any

The conversation, if it is not REQUEST's text.

None

Returns:

Type Description
GenerationRequest

REQUEST with the tools, validated.

frame

frame(data: dict[str, Any]) -> bytes

Write one SSE frame with event: and data:.

Parameters:

Name Type Description Default
data dict[str, Any]

The event, its type the frame's event.

required

Returns:

Type Description
bytes

The frame.

events

events(payloads: list[dict[str, Any]]) -> Reply

Build a streamed reply of the given events.

Parameters:

Name Type Description Default
payloads list[dict[str, Any]]

The events, in order.

required

Returns:

Type Description
Reply

The reply, as text/event-stream.

concurrently async

concurrently(
    client: Any, mode: str, calls: list[dict[str, Any]]
) -> None

Run one call per options at once on the client's one resource.

Parameters:

Name Type Description Default
client Any

The client.

required
mode str

create or stream.

required
calls list[dict[str, Any]]

The provider options of each call.

required

text_of

text_of(received: list[Any]) -> list[str]

Take the text deltas among shared events.

Parameters:

Name Type Description Default
received list[Any]

The events.

required

Returns:

Type Description
list[str]

The text of each TextDeltaEvent.

endless

endless(case: GenerationCase) -> Reply

Build a stream of text deltas that never ends.

Parameters:

Name Type Description Default
case GenerationCase

The provider.

required

Returns:

Type Description
Reply

The reply: the first event, then the first delta over and over.

opener

opener(
    client: Any, case: GenerationCase, mode: str
) -> AbstractAsyncContextManager[Stream[Any]]

Open the raw provider stream or the shared generation stream.

Parameters:

Name Type Description Default
client Any

The client.

required
case GenerationCase

The provider.

required
mode str

raw or generation.

required

Returns:

Type Description
AbstractAsyncContextManager[Stream[Any]]

The stream's context manager.

files

Scenarios every provider's client.files passes.

A provider's tests subclass FilesContract with a FilesCase of its Files API; every scenario then runs against the provider's client on the local server (aiollm.testing.plugin fixtures), with the provider's documented results. Only the wire format differs per provider: the shared results and errors must not. The client needs generation too, since a stored file goes into a turn.

Example

from aiollm.testing.contract import FilesCase, FilesContract class TestExampleFiles(FilesContract): ... case = FilesCase(provider=PROVIDER, client=ExampleClient, ...)

Classes:

Name Description
FilesCase

One provider in the files scenarios.

FilesContract

The files scenarios, for a subclass to run on its provider.

Attributes:

Name Type Description
PDF Final

Bytes the upload sends; the server never looks into them.

FOREIGN Final

A file of a provider no client is of.

CONTAINER_ID Final

The container of the file of a container in the scenarios.

CONTAINER_FILE_ID Final

The id of that file in its container.

PDF module-attribute

PDF: Final = b'%PDF-1.4\n%%EOF\n'

Bytes the upload sends; the server never looks into them.

FOREIGN module-attribute

FOREIGN: Final = ProviderFile(
    provider=Provider("another"), id="file-1"
)

A file of a provider no client is of.

CONTAINER_ID module-attribute

CONTAINER_ID: Final = 'cntr_1'

The container of the file of a container in the scenarios.

CONTAINER_FILE_ID module-attribute

CONTAINER_FILE_ID: Final = 'cfile_1'

The id of that file in its container.

FilesCase dataclass

FilesCase(
    provider: Provider,
    client: Callable[..., Any],
    factory: SessionFactory[Any, Any, ...],
    raw_type: type,
    file: dict[str, Any],
    size: int,
    created_at: datetime,
    expires_at: datetime,
    deleted: dict[str, Any],
    page: dict[str, Any],
    cursor: tuple[str, str],
    last_page: dict[str, Any],
    upload_parts: dict[str, bytes],
    bare_upload_parts: list[str],
    options: tuple[dict[str, Any], dict[str, bytes]] | None,
    message_path: str,
    message: dict[str, Any],
    input_field: str,
    stored_parts: list[dict[str, Any]],
    container_download: str | None,
)

One provider in the files scenarios.

Attributes:

Name Type Description
provider Provider

The provider.

client Callable[..., Any]

Builds the client from a token.

factory SessionFactory[Any, Any, ...]

The session factory of the client, sent to the local server by replacing its api.

raw_type type

The provider's file type, StoredFile.raw.

file dict[str, Any]

A documented file.

size int

Its size in bytes.

created_at datetime

When it was stored, in UTC.

expires_at datetime

When it expires, in UTC.

deleted dict[str, Any]

A documented deletion.

page dict[str, Any]

A page of the list with file and a next page.

cursor tuple[str, str]

The query key and value that ask for the next page.

last_page dict[str, Any]

The body of a page with no next one, its first file another than file.

upload_parts dict[str, bytes]

The form fields of an upload of report.pdf that expires in an hour, in order, besides file.

bare_upload_parts list[str]

The names of the form fields of an upload with neither expiry nor options, in order.

options tuple[dict[str, Any], dict[str, bytes]] | None

Valid upload options, with the fields they change; None if the provider's uploads take none.

message_path str

Path of the generation endpoint.

message dict[str, Any]

A finished generation response.

input_field str

The body field of the conversation.

stored_parts list[dict[str, Any]]

What a document and an image of file become in a user turn.

container_download str | None

The path that downloads file CONTAINER_FILE_ID of container CONTAINER_ID; None for a provider that keeps no files in containers, which refuses the download before sending.

FilesContract

The files scenarios, for a subclass to run on its provider.

A subclass whose name starts with Test sets case; pytest collects its scenarios. It needs the aiollm.testing.plugin fixtures (recorder, api).

Attributes:

Name Type Description
case FilesCase

The provider the scenarios run on.

Methods:

Name Description
client

Build the provider's client, sent to the local server.

ref

Refer to the documented file.

container_file

Refer to a file of a container of the case's provider.

test_an_upload_returns_the_shared_metadata

An upload returns the shared metadata.

test_an_upload_without_expiry_is_kept

An upload without expiry is kept.

test_upload_options_reach_the_form

Upload options reach the form.

test_empty_upload_options_change_nothing

Empty upload options change nothing.

test_an_expiry_out_of_range_is_refused_before_sending

An expiry out of range is refused before sending.

test_an_upload_that_is_not_a_file_is_refused

An upload that is not a file is refused.

test_an_upload_is_not_retried

An upload is not retried.

test_retrieve_reads_the_metadata

Retrieve reads the metadata.

test_list_goes_through_every_page

List goes through every page.

test_list_stops_asking_when_the_caller_stops

List stops asking when the caller stops.

test_delete_returns_nothing

Delete returns nothing.

test_a_missing_file_raises_not_found

A missing file raises not found.

test_download_streams_the_content

Download streams the content.

test_another_providers_file_is_refused_before_sending

Another providers file is refused before sending.

test_a_stored_file_is_sent_by_its_id

A stored file is sent by its id.

test_another_providers_file_in_a_part_is_refused

Another providers file in a part is refused.

test_a_file_of_a_container_is_downloaded_from_it

A file of a container is downloaded from its container.

test_a_file_of_a_container_is_only_downloaded

A file of a container is refused by all but the download.

test_a_file_of_a_container_in_a_part_is_refused

A file of a container in a part is refused.

client

client(api: LocalAPI, monkeypatch: MonkeyPatch) -> Any

Build the provider's client, sent to the local server.

Parameters:

Name Type Description Default
api LocalAPI

The local server's API.

required
monkeypatch MonkeyPatch

pytest's monkeypatch.

required

Returns:

Type Description
Any

The client, built from TEST_TOKEN.

ref

ref() -> ProviderFile

Refer to the documented file.

Returns:

Type Description
ProviderFile

The file of the case's provider.

container_file

container_file() -> ProducedFile

Refer to a file of a container of the case's provider.

Returns:

Type Description
ProducedFile

File CONTAINER_FILE_ID of container CONTAINER_ID.

test_an_upload_returns_the_shared_metadata async

test_an_upload_returns_the_shared_metadata(
    client: Any, recorder: Recorder
) -> None

An upload returns the shared metadata.

test_an_upload_without_expiry_is_kept async

test_an_upload_without_expiry_is_kept(
    client: Any, recorder: Recorder
) -> None

An upload without expiry is kept.

test_upload_options_reach_the_form async

test_upload_options_reach_the_form(
    client: Any, recorder: Recorder
) -> None

Upload options reach the form.

test_empty_upload_options_change_nothing async

test_empty_upload_options_change_nothing(
    client: Any, recorder: Recorder
) -> None

Empty upload options change nothing.

test_an_expiry_out_of_range_is_refused_before_sending async

test_an_expiry_out_of_range_is_refused_before_sending(
    client: Any, recorder: Recorder, seconds: int
) -> None

An expiry out of range is refused before sending.

test_an_upload_that_is_not_a_file_is_refused async

test_an_upload_that_is_not_a_file_is_refused(
    client: Any, recorder: Recorder
) -> None

An upload that is not a file is refused.

test_an_upload_is_not_retried async

test_an_upload_is_not_retried(
    client: Any, recorder: Recorder
) -> None

An upload is not retried.

test_retrieve_reads_the_metadata async

test_retrieve_reads_the_metadata(
    client: Any, recorder: Recorder
) -> None

Retrieve reads the metadata.

test_list_goes_through_every_page async

test_list_goes_through_every_page(
    client: Any, recorder: Recorder
) -> None

List goes through every page.

test_list_stops_asking_when_the_caller_stops async

test_list_stops_asking_when_the_caller_stops(
    client: Any, recorder: Recorder
) -> None

List stops asking when the caller stops.

test_delete_returns_nothing async

test_delete_returns_nothing(
    client: Any, recorder: Recorder
) -> None

Delete returns nothing.

test_a_missing_file_raises_not_found async

test_a_missing_file_raises_not_found(
    client: Any, recorder: Recorder
) -> None

A missing file raises not found.

test_download_streams_the_content async

test_download_streams_the_content(
    client: Any, recorder: Recorder, tmp_path: Path
) -> None

Download streams the content.

test_another_providers_file_is_refused_before_sending async

test_another_providers_file_is_refused_before_sending(
    client: Any, recorder: Recorder, call: str
) -> None

Another providers file is refused before sending.

test_a_stored_file_is_sent_by_its_id async

test_a_stored_file_is_sent_by_its_id(
    client: Any, recorder: Recorder
) -> None

A stored file is sent by its id.

test_another_providers_file_in_a_part_is_refused async

test_another_providers_file_in_a_part_is_refused(
    client: Any, recorder: Recorder
) -> None

Another providers file in a part is refused.

test_a_file_of_a_container_is_downloaded_from_it async

test_a_file_of_a_container_is_downloaded_from_it(
    client: Any, recorder: Recorder, tmp_path: Path
) -> None

A file of a container is downloaded from its container.

test_a_file_of_a_container_is_only_downloaded async

test_a_file_of_a_container_is_only_downloaded(
    client: Any, recorder: Recorder, call: str
) -> None

A file of a container is refused by all but the download.

test_a_file_of_a_container_in_a_part_is_refused async

test_a_file_of_a_container_in_a_part_is_refused(
    client: Any, recorder: Recorder
) -> None

A file of a container in a part is refused.

clients

Scenarios every provider client passes.

A provider's tests subclass ClientContract with a ClientCase of its client: how it is built from a token, a key in the environment or a given session, and the warnings about a session that does not fit it.

Example

from aiollm.testing.contract import ClientCase, ClientContract class TestExampleClient(ClientContract): ... case = ClientCase(client=ExampleClient, provider=PROVIDER, ...)

Classes:

Name Description
AnotherErrorResolver

The resolver of a provider no client is of.

ClientCase

One provider client in the client scenarios.

ClientContract

The client scenarios, for a subclass to run on its provider.

Attributes:

Name Type Description
LIBRARY Final

The library's directory, which a warning skips.

LIBRARY module-attribute

LIBRARY: Final = Path(aiollm.__file__).resolve().parent

The library's directory, which a warning skips.

AnotherErrorResolver

Bases: BaseErrorResolver

The resolver of a provider no client is of.

The default ClientCase.foreign_resolver: a session with it does not fit any client.

ClientCase dataclass

ClientCase(
    client: type[BaseProviderClient],
    provider: Provider,
    api: IAPI,
    url: tuple[str, str],
    resolver: type[BaseErrorResolver],
    env: str,
    auths: tuple[IAuth, ...] = (BearerAuth("k"),),
    foreign_resolver: type[
        BaseErrorResolver
    ] = AnotherErrorResolver,
)

One provider client in the client scenarios.

Attributes:

Name Type Description
client type[BaseProviderClient]

The client class.

provider Provider

Its provider.

api IAPI

The API its sessions are built for.

url tuple[str, str]

A path of the API and the URL it builds.

resolver type[BaseErrorResolver]

The provider's error resolver class.

env str

The environment variable with the key.

auths tuple[IAuth, ...]

Authentications the provider takes.

foreign_resolver type[BaseErrorResolver]

Another provider's resolver, warned about.

ClientContract

The client scenarios, for a subclass to run on its provider.

A subclass whose name starts with Test sets case; pytest collects its scenarios.

Attributes:

Name Type Description
case ClientCase

The client the scenarios run on.

Methods:

Name Description
test_client_from_a_token_outside_a_coroutine

Client from a token outside a coroutine.

test_api_url

Api url.

test_client_needs_a_token_or_a_session

Client needs a token or a session.

test_client_reads_the_key_from_the_environment

Client reads the key from the environment.

test_explicit_token_wins_over_the_environment

Explicit token wins over the environment.

test_session_ignores_the_environment

Session ignores the environment.

test_missing_key_names_the_variable

Missing key names the variable.

test_a_fitting_session_gets_no_warning

A fitting session gets no warning.

test_a_resolver_of_no_or_another_provider_is_warned_about

A resolver of no or another provider is warned about.

test_another_auth_header_is_warned_about

Another auth header is warned about.

test_other_sessions_are_not_checked

Other sessions are not checked.

test_session_arguments_are_refused_with_a_session

Session arguments are refused with a session.

test_client_from_a_token_outside_a_coroutine

test_client_from_a_token_outside_a_coroutine() -> None

Client from a token outside a coroutine.

test_api_url

test_api_url() -> None

Api url.

test_client_needs_a_token_or_a_session

test_client_needs_a_token_or_a_session(kwargs: Any) -> None

Client needs a token or a session.

test_client_reads_the_key_from_the_environment

test_client_reads_the_key_from_the_environment(
    monkeypatch: MonkeyPatch,
) -> None

Client reads the key from the environment.

test_explicit_token_wins_over_the_environment

test_explicit_token_wins_over_the_environment(
    monkeypatch: MonkeyPatch,
) -> None

Explicit token wins over the environment.

test_session_ignores_the_environment

test_session_ignores_the_environment(
    monkeypatch: MonkeyPatch,
) -> None

Session ignores the environment.

test_missing_key_names_the_variable

test_missing_key_names_the_variable(
    monkeypatch: MonkeyPatch, value: str | None
) -> None

Missing key names the variable.

test_a_fitting_session_gets_no_warning

test_a_fitting_session_gets_no_warning() -> None

A fitting session gets no warning.

test_a_resolver_of_no_or_another_provider_is_warned_about

test_a_resolver_of_no_or_another_provider_is_warned_about(
    resolver: str,
) -> None

A resolver of no or another provider is warned about.

test_another_auth_header_is_warned_about

test_another_auth_header_is_warned_about() -> None

Another auth header is warned about.

test_other_sessions_are_not_checked

test_other_sessions_are_not_checked() -> None

Other sessions are not checked.

test_session_arguments_are_refused_with_a_session

test_session_arguments_are_refused_with_a_session() -> None

Session arguments are refused with a session.

extension

The extension API: what a provider package may take from the core.

extension_api.json, next to this module, lists by module every name a provider package imports from aiollm. A provider's tests call check_extension_api with its package: an import of any other name of the core, of another provider or a plain import aiollm... is reported, so a provider learns at once when it leans on something the core does not promise to keep. The core removes or changes a listed name only with a Breaking changelog fragment.

Example

def test_the_package_takes_only_the_extension_api() -> None: ... assert check_extension_api("aiollm_example") == []

Functions:

Name Description
outside_names

Find the imports of a provider module that the extension API lacks.

check_extension_api

Check every module of a provider package against the extension API.

Attributes:

Name Type Description
EXTENSION_API Final[Mapping[str, Sequence[str]]]

Every name of the extension API, by the module it is imported from.

PROVIDER_PREFIXES Final

Module prefixes of provider packages, which never import each other.

EXTENSION_API module-attribute

EXTENSION_API: Final[Mapping[str, Sequence[str]]] = (
    json.loads(
        Path(__file__)
        .with_name("extension_api.json")
        .read_text(encoding="utf-8")
    )
)

Every name of the extension API, by the module it is imported from.

PROVIDER_PREFIXES module-attribute

PROVIDER_PREFIXES: Final = ('aiollm_',)

Module prefixes of provider packages, which never import each other.

outside_names

outside_names(
    package: str,
    source: str,
    allowed: Mapping[str, Sequence[str]] = EXTENSION_API,
) -> list[str]

Find the imports of a provider module that the extension API lacks.

Parameters:

Name Type Description Default
package str

The provider package the module belongs to, such as aiollm_openai; its own modules may be imported.

required
source str

The module's source.

required
allowed Mapping[str, Sequence[str]]

The extension API, by module.

EXTENSION_API

Returns:

Type Description
list[str]

Each wrong import as line: module.name, or ``line: import

list[str]

module`` for a plain import of the core.

check_extension_api

check_extension_api(
    package: str,
    allowed: Mapping[str, Sequence[str]] = EXTENSION_API,
) -> list[str]

Check every module of a provider package against the extension API.

Parameters:

Name Type Description Default
package str

The importable name of the provider package, such as aiollm_openai.

required
allowed Mapping[str, Sequence[str]]

The extension API, by module.

EXTENSION_API

Returns:

Type Description
list[str]

Each wrong import as path:line: module.name, the path

list[str]

relative to the package; empty when the package takes only the

list[str]

extension API.

Raises:

Type Description
ValueError

The package has no source files of its own, such as a namespace package.

wire

Checks of a provider's types against the snapshot of its upstream schemas.

A provider package keeps a snapshot of the schemas its provider publishes (an official OpenAPI file, or the types of an official SDK): one folder per endpoint, <provider>/<endpoint>, with request.schema.json, response.schema.json, an optional stream.schema.json and the documented examples responses.json and events.json. These functions read such a snapshot and compare the provider's generated models with it, so the library drifting from the snapshot fails a test; the provider drifting from the library shows once the snapshot is refreshed. It needs jsonschema, which the testing extra installs.

Example

snapshot = Snapshot(Path("tests/contracts"), "example/chat") errors(snapshot.validator("request.schema.json"), body) [] field_problems([snapshot], models, params=False) []

Classes:

Name Description
Snapshot

The snapshot of one endpoint's upstream schemas and examples.

Functions:

Name Description
closed

Close every object of a schema that lists its properties.

errors

List the schema errors of a value with their paths.

snapshot_schemas

Read the schemas of several snapshots that exist.

upstream_definitions

Collect the definitions of endpoints' upstream schemas.

resolve

Follow a $ref to its definition.

allows_null

Tell whether a schema node accepts null.

provider_models

Find the models of a provider.

upstream_names

Read the upstream definitions a model declares it mirrors.

is_constant

Tell a single-value tag that defaults to its one value.

naming_problems

Check that every model names the upstream definition it mirrors.

field_problems

Compare every field of the models with its upstream definition.

Attributes:

Name Type Description
REQUEST_SCHEMA Final

The request schema of an endpoint, closed when it is read.

RESPONSE_FILES Final

The upstream schemas of the response and event models.

REQUEST_FILES Final

The upstream schema of the request parameter models.

REQUEST_SCHEMA module-attribute

REQUEST_SCHEMA: Final = 'request.schema.json'

The request schema of an endpoint, closed when it is read.

RESPONSE_FILES module-attribute

RESPONSE_FILES: Final = (
    "response.schema.json",
    "stream.schema.json",
)

The upstream schemas of the response and event models.

REQUEST_FILES module-attribute

REQUEST_FILES: Final = (REQUEST_SCHEMA,)

The upstream schema of the request parameter models.

Snapshot dataclass

Snapshot(root: Path, name: str)

The snapshot of one endpoint's upstream schemas and examples.

Attributes:

Name Type Description
root Path

The folder of every snapshot of the package.

name str

The endpoint's folder in it, <provider>/<endpoint>.

Methods:

Name Description
has

Tell whether the snapshot has a file.

load

Read one file of the snapshot.

validator

Build a validator of a schema of the snapshot.

examples

Take the documented examples of the snapshot.

has

has(file: str) -> bool

Tell whether the snapshot has a file.

Parameters:

Name Type Description Default
file str

The file name, such as stream.schema.json.

required

Returns:

Type Description
bool

Whether it exists; an endpoint without a stream has no

bool

stream schema.

load

load(file: str) -> Any

Read one file of the snapshot.

Parameters:

Name Type Description Default
file str

The file name.

required

Returns:

Type Description
Any

The decoded JSON.

validator

validator(file: str) -> Draft202012Validator

Build a validator of a schema of the snapshot.

Parameters:

Name Type Description Default
file str

The schema's file name.

required

Returns:

Type Description
Draft202012Validator

The validator; the request schema is closed (see closed).

examples

examples(file: str) -> list[Any]

Take the documented examples of the snapshot.

Parameters:

Name Type Description Default
file str

responses.json or events.json, a list of {"body": ...} entries.

required

Returns:

Type Description
list[Any]

The example bodies.

closed

closed(schema: Any) -> Any

Close every object of a schema that lists its properties.

The upstream request schemas leave additionalProperties out, which JSON Schema reads as "anything goes", while the APIs reject unknown fields.

Parameters:

Name Type Description Default
schema Any

A schema or a node of one.

required

Returns:

Type Description
Any

A copy with additionalProperties: false wherever an object

Any

has properties and does not say otherwise.

errors

errors(
    validator: Draft202012Validator, value: Any
) -> list[str]

List the schema errors of a value with their paths.

Parameters:

Name Type Description Default
validator Draft202012Validator

The schema's validator.

required
value Any

The value.

required

Returns:

Type Description
list[str]

One path: message line per error; empty for a valid value.

snapshot_schemas

snapshot_schemas(
    snapshots: Iterable[Snapshot], files: Sequence[str]
) -> Iterator[dict[str, Any]]

Read the schemas of several snapshots that exist.

Parameters:

Name Type Description Default
snapshots Iterable[Snapshot]

The endpoints.

required
files Sequence[str]

Schema file names; one a snapshot lacks is skipped.

required

Yields:

Type Description
dict[str, Any]

Each schema.

upstream_definitions

upstream_definitions(
    snapshots: Iterable[Snapshot],
    files: Sequence[str] = RESPONSE_FILES,
) -> dict[str, Any]

Collect the definitions of endpoints' upstream schemas.

Parameters:

Name Type Description Default
snapshots Iterable[Snapshot]

The endpoints.

required
files Sequence[str]

The schemas to read: the response and stream ones, or the request one.

RESPONSE_FILES

Returns:

Type Description
dict[str, Any]

Every $defs entry of the schemas, and each root schema under

dict[str, Any]

its title (such as Response), and every object written

dict[str, Any]

inline in a property under <Owner>.<property>; a nullable

dict[str, Any]

object (anyOf of it and null) is the object.

resolve

resolve(node: Any, definitions: dict[str, Any]) -> Any

Follow a $ref to its definition.

Parameters:

Name Type Description Default
node Any

A schema node.

required
definitions dict[str, Any]

The definitions by name.

required

Returns:

Type Description
Any

The referenced definition, or the node itself.

allows_null

allows_null(node: Any, definitions: dict[str, Any]) -> bool

Tell whether a schema node accepts null.

Parameters:

Name Type Description Default
node Any

A schema node.

required
definitions dict[str, Any]

The definitions by name.

required

Returns:

Type Description
bool

Whether null is valid for the node.

provider_models

provider_models(
    modules: Iterable[ModuleType],
    covered: Iterable[type[BaseModel]] = (),
) -> Iterator[type[BaseAPIType]]

Find the models of a provider.

Parameters:

Name Type Description Default
modules Iterable[ModuleType]

The modules of the models, such as every module of the provider's types package.

required
covered Iterable[type[BaseModel]]

Methods whose own fields a test checks otherwise; the parameter models they inherit from are left out.

()

Yields:

Type Description
type[BaseAPIType]

Every BaseAPIType subclass defined in the modules, except the

type[BaseAPIType]

BaseUnknown ones, which keep whatever an unknown type

type[BaseAPIType]

sends, and the bases of covered.

upstream_names

upstream_names(model: type[BaseAPIType]) -> tuple[str, ...]

Read the upstream definitions a model declares it mirrors.

Parameters:

Name Type Description Default
model type[BaseAPIType]

A provider model with __upstream_schema__, one name or several.

required

Returns:

Type Description
tuple[str, ...]

The names.

is_constant

is_constant(annotation: Any, default: Any) -> bool

Tell a single-value tag that defaults to its one value.

Such a field (type: Literal["function"] = "function") is always sent, so a schema that requires it is met without the caller.

Parameters:

Name Type Description Default
annotation Any

The field's type.

required
default Any

Its default.

required

Returns:

Type Description
bool

Whether the type is a one-value Literal of the default.

naming_problems

naming_problems(
    snapshots: Sequence[Snapshot],
    models: Sequence[type[BaseAPIType]],
    *,
    params: bool,
) -> list[str]

Check that every model names the upstream definition it mirrors.

Parameters:

Name Type Description Default
snapshots Sequence[Snapshot]

The endpoints the models serve.

required
models Sequence[type[BaseAPIType]]

The models, from provider_models.

required
params bool

Whether they are request parameter models, checked against the request schema, or response and event models.

required

Returns:

Type Description
list[str]

One line per model without __upstream_schema__ or naming a

list[str]

definition the schemas lack; empty when all is well.

field_problems

field_problems(
    snapshots: Sequence[Snapshot],
    models: Iterable[type[BaseAPIType]],
    *,
    params: bool,
) -> list[str]

Compare every field of the models with its upstream definition.

Validating examples sees only the values the docs happen to show; this compares every modeled field with the schema itself. A response field may only be laxer than the schema, to read what is sent; a request field follows it both ways: None only where null is allowed, required only where it is (a one-value tag that defaults to its value counts as given).

Parameters:

Name Type Description Default
snapshots Sequence[Snapshot]

The endpoints the models serve.

required
models Iterable[type[BaseAPIType]]

The models, each with __upstream_schema__.

required
params bool

Whether they are request parameter models.

required

Returns:

Type Description
list[str]

One line per mismatch; empty when all is well.

echo

The echo provider: a provider package made for tests.

Its API exists only on the local server a test scripts, so it runs every shared scenario of aiollm.testing.contract without a real provider, and it takes nothing from the core outside the extension API: it is the smallest example of a provider package. Its client binds the aiohttp transport, as a provider package's does.

Modules:

Name Description
api

URL and provider of the echo API.

client

Client of the echo API.

files

Adapter of shared files to the echo API.

generation

Adapter of shared generation to the echo API.

methods

Methods of the echo API: generation, files and container files.

resolver

Error mapping of the echo API.

types

Types of the echo API: its response, stream events, files and options.

Classes:

Name Description
EchoAPI

Builds echo request URLs.

EchoClient

Client of the echo API, a provider made for tests.

EchoFilesAdapter

Stores files with the echo /files.

EchoGenerationAdapter

Generates text with POST /echo.

EchoErrorResolver

Error resolver of the echo API.

Attributes:

Name Type Description
ECHO_API Final[EchoAPI]

The echo API, version 1.

PROVIDER Final

The provider of this package; its client, resolver and adapters use it.

session_factory Final

Builds the AiohttpSession of an EchoClient from a token.

ECHO_API module-attribute

ECHO_API: Final[EchoAPI] = EchoAPI(
    base_url="https://echo.invalid/", version="v1"
)

The echo API, version 1.

.invalid never resolves, so a client that was not pointed at a local server fails at once instead of reaching anything.

PROVIDER module-attribute

PROVIDER: Final = Provider('echo')

The provider of this package; its client, resolver and adapters use it.

session_factory module-attribute

session_factory: Final = SessionFactory(
    AiohttpSession,
    auth_factory=BearerAuth,
    api=ECHO_API,
    error_resolver=EchoErrorResolver(),
)

Builds the AiohttpSession of an EchoClient from a token.

EchoAPI

EchoAPI(base_url: str | URL, version: str)

Bases: BaseAPI

Builds echo request URLs.

EchoClient

EchoClient(
    token: str | None = None,
    *,
    session: ISession | None = None,
    **session_kwargs: Unpack[AiohttpSessionOptions],
)

Bases: BaseProviderClient

Client of the echo API, a provider made for tests.

The echo API exists only on a local server that a test scripts (aiollm.testing.aiohttp.Recorder): it is the smallest provider that every shared scenario runs on, built from the extension API alone, and an example of a provider package.

Attributes:

Name Type Description
generation

Shared text generation over POST /echo.

files

Shared file storage over /files.

Example

monkeypatch.setattr(session_factory, "api", api) async with EchoClient(TEST_TOKEN) as client: ... response = await client.generation.create( ... model="echo-1", input="Hi" ... )

Create the client from a token or a ready session.

A provider client declares the arguments of its session, so a misspelled one is a type error.

Parameters:

Name Type Description Default
token str | None

API key; an AiohttpSession with the echo API's URL, error resolver and BearerAuth is created from it. Without a token and a session, ECHO_API_KEY is used.

None
session ISession | None

A ready session, for custom settings; it is checked with warn_about_session.

None
**session_kwargs Unpack[AiohttpSessionOptions]

Further arguments of the AiohttpSession built from the token, such as timeout or max_body_size.

{}

Raises:

Type Description
ValueError

If both token and session are given, or neither is and ECHO_API_KEY is unset or empty, or session_kwargs are given with a session.

EchoFilesAdapter

EchoFilesAdapter(*, page_size: int = DEFAULT_PAGE_SIZE)

Bases: BaseFilesAdapter[EchoFile, EchoFilePage]

Stores files with the echo /files.

expires_in_seconds is sent as it is, and provider_options may add a purpose. The list goes newest first, page_size files a request, by the page's next.

Attributes:

Name Type Description
page_size

Files asked for in one page of the list.

Example

files = FilesResource(session, EchoFilesAdapter())

Store the page size.

Parameters:

Name Type Description Default
page_size int

Files asked for in one page of the list.

DEFAULT_PAGE_SIZE

Methods:

Name Description
with_options

Return a copy that sends the options with every upload.

to_upload_method

Build the method that stores a file.

to_retrieve_method

Build the method that reads a file's metadata.

to_list_method

Build the method that reads one page of the stored files.

to_page

Take the files of a page and where the next one starts.

to_delete_method

Build the method that deletes a file.

to_download_method

Build the method that streams a file's content.

to_stored_file

Convert the echo file into the shared metadata.

with_options

with_options(options: EchoFileOptions) -> Self

Return a copy that sends the options with every upload.

Parameters:

Name Type Description Default
options EchoFileOptions

Fields of UploadFile the shared upload does not hold, such as {"purpose": "vision"}.

required

Returns:

Type Description
Self

A copy of the adapter; this one is left as it is.

Raises:

Type Description
RequestValidationError

A key is not one of the options.

to_upload_method

to_upload_method(request: FileUpload) -> UploadFile

Build the method that stores a file.

Parameters:

Name Type Description Default
request FileUpload

The shared upload.

required

Returns:

Type Description
UploadFile

UploadFile with the file, its expiry and the options.

Raises:

Type Description
RequestValidationError

The echo API cannot take the upload, such as an expiry outside an hour to thirty days.

to_retrieve_method

to_retrieve_method(file_id: str) -> RetrieveFile

Build the method that reads a file's metadata.

Parameters:

Name Type Description Default
file_id str

The file's id.

required

Returns:

Type Description
RetrieveFile

RetrieveFile.

to_list_method

to_list_method(cursor: str | None) -> ListFiles

Build the method that reads one page of the stored files.

Parameters:

Name Type Description Default
cursor str | None

next of the page before, or None for the first page.

required

Returns:

Type Description
ListFiles

ListFiles of page_size files.

to_page

to_page(
    raw: EchoFilePage,
) -> tuple[Sequence[EchoFile], str | None]

Take the files of a page and where the next one starts.

Parameters:

Name Type Description Default
raw EchoFilePage

The page.

required

Returns:

Type Description
tuple[Sequence[EchoFile], str | None]

Its files and its next.

to_delete_method

to_delete_method(file_id: str) -> DeleteFile

Build the method that deletes a file.

Parameters:

Name Type Description Default
file_id str

The file's id.

required

Returns:

Type Description
DeleteFile

DeleteFile.

to_download_method

to_download_method(
    file: ProviderFile,
) -> DownloadFile | DownloadContainerFile

Build the method that streams a file's content.

Parameters:

Name Type Description Default
file ProviderFile

The file.

required

Returns:

Type Description
DownloadFile | DownloadContainerFile

DownloadContainerFile for a file of a container, else

DownloadFile | DownloadContainerFile

DownloadFile.

to_stored_file

to_stored_file(raw: EchoFile) -> StoredFile[EchoFile]

Convert the echo file into the shared metadata.

Parameters:

Name Type Description Default
raw EchoFile

The echo file.

required

Returns:

Type Description
StoredFile[EchoFile]

Its id, name, bytes as size and its times in UTC.

EchoGenerationAdapter

EchoGenerationAdapter()

Bases: BaseCompositeAdapter[GenerationRequest, EchoResponse, GenerationResponse[EchoResponse], EchoEvent, GenerationEvent[EchoEvent]]

Generates text with POST /echo.

Every shared field is sent under its own name: the effort as its value, the tools as name, description, parameters and strict, the tool choice as its value or {"name": ...}. A request the conversation rules refuse raises RequestValidationError before anything is sent.

Example

generation = GenerationResource(session, EchoGenerationAdapter())

Start without provider options.

Methods:

Name Description
with_options

Return a copy that sends the options with every method.

to_method

Build the method that generates at once.

to_response

Convert a finished generation.

to_stream_method

Build the method that streams.

to_events

Convert the events of a stream.

with_options

with_options(options: EchoGenerationOptions) -> Self

Return a copy that sends the options with every method.

Parameters:

Name Type Description Default
options EchoGenerationOptions

Fields of Generate the shared request does not hold, such as {"temperature": 0.2}.

required

Returns:

Type Description
Self

A copy of the adapter; this one is left as it is.

Raises:

Type Description
RequestValidationError

A key is not one of the options.

to_method

to_method(request: GenerationRequest) -> Generate

Build the method that generates at once.

Parameters:

Name Type Description Default
request GenerationRequest

The shared request.

required

Returns:

Type Description
Generate

Generate for the request.

Raises:

Type Description
RequestValidationError

The echo API cannot take the request.

to_response

Convert a finished generation.

Parameters:

Name Type Description Default
raw EchoResponse

The echo response.

required
request GenerationRequest

The request it answers.

required

Returns:

Type Description
GenerationResponse[EchoResponse]

Its text, calls, counts and stop, the output as native.

to_stream_method

to_stream_method(
    request: GenerationRequest,
) -> StreamGenerate

Build the method that streams.

Parameters:

Name Type Description Default
request GenerationRequest

The shared request.

required

Returns:

Type Description
StreamGenerate

StreamGenerate for the request.

Raises:

Type Description
RequestValidationError

The echo API cannot take the request.

to_events async

Convert the events of a stream.

Text deltas become TextDeltaEvent; the events of a call ToolCallStartEvent, ToolCallDeltaEvent and ToolCallDoneEvent; those of a built-in tool call ServerToolCallEvent; done becomes CompletedEvent with what the stream told, the built-in calls, citations and files among it; every other event is an OtherEvent.

Parameters:

Name Type Description Default
items AsyncIterator[EchoEvent]

The events the echo API sent.

required
request GenerationRequest

The request the stream answers.

required

Yields:

Type Description
AsyncGenerator[GenerationEvent[EchoEvent]]

One shared event per echo event.

Raises:

Type Description
APIResponseError

done came without start, or a call event names a call no tool_call_start opened.

EchoErrorResolver

Bases: BaseErrorResolver

Error resolver of the echo API.

Errors come in the common {"error": {"code", "message"}} shape, over HTTP and as the error event of a stream. An exhausted quota (insufficient_quota) is a QuotaExceededError and is never retried; everything else is classified by its status.

Methods:

Name Description
classify_status

Pick the error class and transience of the echo statuses.

classify_status

classify_status(
    *,
    status: int,
    code: str | None,
    error_type: str | None,
    message: str | None,
    retry_after: float | None,
) -> tuple[type[APIStatusError], bool]

Pick the error class and transience of the echo statuses.

Parameters:

Name Type Description Default
status int

HTTP status the error is classified by.

required
code str | None

Provider error code, if any.

required
error_type str | None

Provider error type, if any.

required
message str | None

Error message, if any.

required
retry_after float | None

Seconds from Retry-After, if any.

required

Returns:

Type Description
type[APIStatusError]

QuotaExceededError (never transient) for an exhausted

bool

quota, else the default classification.

api

URL and provider of the echo API.

Classes:

Name Description
EchoAPI

Builds echo request URLs.

Attributes:

Name Type Description
PROVIDER Final

The provider of this package; its client, resolver and adapters use it.

ECHO_API Final[EchoAPI]

The echo API, version 1.

PROVIDER module-attribute

PROVIDER: Final = Provider('echo')

The provider of this package; its client, resolver and adapters use it.

ECHO_API module-attribute

ECHO_API: Final[EchoAPI] = EchoAPI(
    base_url="https://echo.invalid/", version="v1"
)

The echo API, version 1.

.invalid never resolves, so a client that was not pointed at a local server fails at once instead of reaching anything.

EchoAPI

EchoAPI(base_url: str | URL, version: str)

Bases: BaseAPI

Builds echo request URLs.

client

Client of the echo API.

Classes:

Name Description
EchoClient

Client of the echo API, a provider made for tests.

Attributes:

Name Type Description
session_factory Final

Builds the AiohttpSession of an EchoClient from a token.

ECHO_API_KEY_ENV Final

Environment variable with the key, read without a token or session.

session_factory module-attribute

session_factory: Final = SessionFactory(
    AiohttpSession,
    auth_factory=BearerAuth,
    api=ECHO_API,
    error_resolver=EchoErrorResolver(),
)

Builds the AiohttpSession of an EchoClient from a token.

ECHO_API_KEY_ENV module-attribute

ECHO_API_KEY_ENV: Final = 'ECHO_API_KEY'

Environment variable with the key, read without a token or session.

EchoClient

EchoClient(
    token: str | None = None,
    *,
    session: ISession | None = None,
    **session_kwargs: Unpack[AiohttpSessionOptions],
)

Bases: BaseProviderClient

Client of the echo API, a provider made for tests.

The echo API exists only on a local server that a test scripts (aiollm.testing.aiohttp.Recorder): it is the smallest provider that every shared scenario runs on, built from the extension API alone, and an example of a provider package.

Attributes:

Name Type Description
generation

Shared text generation over POST /echo.

files

Shared file storage over /files.

Example

monkeypatch.setattr(session_factory, "api", api) async with EchoClient(TEST_TOKEN) as client: ... response = await client.generation.create( ... model="echo-1", input="Hi" ... )

Create the client from a token or a ready session.

A provider client declares the arguments of its session, so a misspelled one is a type error.

Parameters:

Name Type Description Default
token str | None

API key; an AiohttpSession with the echo API's URL, error resolver and BearerAuth is created from it. Without a token and a session, ECHO_API_KEY is used.

None
session ISession | None

A ready session, for custom settings; it is checked with warn_about_session.

None
**session_kwargs Unpack[AiohttpSessionOptions]

Further arguments of the AiohttpSession built from the token, such as timeout or max_body_size.

{}

Raises:

Type Description
ValueError

If both token and session are given, or neither is and ECHO_API_KEY is unset or empty, or session_kwargs are given with a session.

generation

Adapter of shared generation to the echo API.

Classes:

Name Description
EchoGenerationAdapter

Generates text with POST /echo.

Functions:

Name Description
to_input

Convert the shared input into the echo input.

to_usage

Convert the echo token counts.

to_stop

Tell how a generation stopped.

to_server_call

Convert a built-in tool call.

to_citation

Convert a source.

to_file

Convert a file a built-in tool made.

Attributes:

Name Type Description
FINISHED Final

Stops of a generation that reached its end.

INCOMPLETE_REASONS Final[Mapping[str, IncompleteReason]]

Shared reason of each early stop that has one; any other is OTHER.

FINISHED module-attribute

FINISHED: Final = frozenset({'end', 'tool_calls'})

Stops of a generation that reached its end.

INCOMPLETE_REASONS module-attribute

INCOMPLETE_REASONS: Final[
    Mapping[str, IncompleteReason]
] = {
    "max_tokens": IncompleteReason.MAX_TOKENS,
    "refusal": IncompleteReason.REFUSAL,
    "paused": IncompleteReason.PAUSED,
}

Shared reason of each early stop that has one; any other is OTHER.

EchoGenerationAdapter

EchoGenerationAdapter()

Bases: BaseCompositeAdapter[GenerationRequest, EchoResponse, GenerationResponse[EchoResponse], EchoEvent, GenerationEvent[EchoEvent]]

Generates text with POST /echo.

Every shared field is sent under its own name: the effort as its value, the tools as name, description, parameters and strict, the tool choice as its value or {"name": ...}. A request the conversation rules refuse raises RequestValidationError before anything is sent.

Example

generation = GenerationResource(session, EchoGenerationAdapter())

Start without provider options.

Methods:

Name Description
with_options

Return a copy that sends the options with every method.

to_method

Build the method that generates at once.

to_response

Convert a finished generation.

to_stream_method

Build the method that streams.

to_events

Convert the events of a stream.

with_options

with_options(options: EchoGenerationOptions) -> Self

Return a copy that sends the options with every method.

Parameters:

Name Type Description Default
options EchoGenerationOptions

Fields of Generate the shared request does not hold, such as {"temperature": 0.2}.

required

Returns:

Type Description
Self

A copy of the adapter; this one is left as it is.

Raises:

Type Description
RequestValidationError

A key is not one of the options.

to_method

to_method(request: GenerationRequest) -> Generate

Build the method that generates at once.

Parameters:

Name Type Description Default
request GenerationRequest

The shared request.

required

Returns:

Type Description
Generate

Generate for the request.

Raises:

Type Description
RequestValidationError

The echo API cannot take the request.

to_response

Convert a finished generation.

Parameters:

Name Type Description Default
raw EchoResponse

The echo response.

required
request GenerationRequest

The request it answers.

required

Returns:

Type Description
GenerationResponse[EchoResponse]

Its text, calls, counts and stop, the output as native.

to_stream_method

to_stream_method(
    request: GenerationRequest,
) -> StreamGenerate

Build the method that streams.

Parameters:

Name Type Description Default
request GenerationRequest

The shared request.

required

Returns:

Type Description
StreamGenerate

StreamGenerate for the request.

Raises:

Type Description
RequestValidationError

The echo API cannot take the request.

to_events async

Convert the events of a stream.

Text deltas become TextDeltaEvent; the events of a call ToolCallStartEvent, ToolCallDeltaEvent and ToolCallDoneEvent; those of a built-in tool call ServerToolCallEvent; done becomes CompletedEvent with what the stream told, the built-in calls, citations and files among it; every other event is an OtherEvent.

Parameters:

Name Type Description Default
items AsyncIterator[EchoEvent]

The events the echo API sent.

required
request GenerationRequest

The request the stream answers.

required

Yields:

Type Description
AsyncGenerator[GenerationEvent[EchoEvent]]

One shared event per echo event.

Raises:

Type Description
APIResponseError

done came without start, or a call event names a call no tool_call_start opened.

to_input

to_input(
    source: str | Sequence[InputItem],
) -> list[dict[str, Any]]

Convert the shared input into the echo input.

A text becomes one user message. A turn, or a turn of calls, with echo's own native output goes back as that output; any other turn of calls becomes an assistant message with its text and tool_calls. A result is a tool message.

Parameters:

Name Type Description Default
source str | Sequence[InputItem]

The text or the conversation of the shared request.

required

Returns:

Type Description
list[dict[str, Any]]

The messages.

to_usage

to_usage(usage: EchoUsage) -> TokenUsage

Convert the echo token counts.

Parameters:

Name Type Description Default
usage EchoUsage

The counts.

required

Returns:

Type Description
TokenUsage

The shared counts, with their total.

to_stop

to_stop(stop: str, refusal: str | None) -> Stop

Tell how a generation stopped.

Parameters:

Name Type Description Default
stop str

The echo stop.

required
refusal str | None

What the model said about a refusal, if anything.

required

Returns:

Type Description
Stop

Nothing set if it reached its end; else the shared and the echo

Stop

reason, and the refusal's text for a refusal.

to_server_call

to_server_call(item: ServerToolItem) -> ServerToolCall

Convert a built-in tool call.

Parameters:

Name Type Description Default
item ServerToolItem

The echo item.

required

Returns:

Type Description
ServerToolCall

The shared call; a kind or status the library lacks is OTHER

ServerToolCall

or FAILED.

to_citation

to_citation(item: CitationItem) -> Citation

Convert a source.

Parameters:

Name Type Description Default
item CitationItem

The echo item.

required

Returns:

Type Description
Citation

The shared citation.

to_file

to_file(item: FileItem) -> ProducedFile

Convert a file a built-in tool made.

Parameters:

Name Type Description Default
item FileItem

The echo item.

required

Returns:

Type Description
ProducedFile

The shared file.

files

Adapter of shared files to the echo API.

Classes:

Name Description
EchoFilesAdapter

Stores files with the echo /files.

Attributes:

Name Type Description
DEFAULT_PAGE_SIZE Final

Files asked for in one page of client.files.list().

DEFAULT_PAGE_SIZE module-attribute

DEFAULT_PAGE_SIZE: Final = 100

Files asked for in one page of client.files.list().

EchoFilesAdapter

EchoFilesAdapter(*, page_size: int = DEFAULT_PAGE_SIZE)

Bases: BaseFilesAdapter[EchoFile, EchoFilePage]

Stores files with the echo /files.

expires_in_seconds is sent as it is, and provider_options may add a purpose. The list goes newest first, page_size files a request, by the page's next.

Attributes:

Name Type Description
page_size

Files asked for in one page of the list.

Example

files = FilesResource(session, EchoFilesAdapter())

Store the page size.

Parameters:

Name Type Description Default
page_size int

Files asked for in one page of the list.

DEFAULT_PAGE_SIZE

Methods:

Name Description
with_options

Return a copy that sends the options with every upload.

to_upload_method

Build the method that stores a file.

to_retrieve_method

Build the method that reads a file's metadata.

to_list_method

Build the method that reads one page of the stored files.

to_page

Take the files of a page and where the next one starts.

to_delete_method

Build the method that deletes a file.

to_download_method

Build the method that streams a file's content.

to_stored_file

Convert the echo file into the shared metadata.

with_options

with_options(options: EchoFileOptions) -> Self

Return a copy that sends the options with every upload.

Parameters:

Name Type Description Default
options EchoFileOptions

Fields of UploadFile the shared upload does not hold, such as {"purpose": "vision"}.

required

Returns:

Type Description
Self

A copy of the adapter; this one is left as it is.

Raises:

Type Description
RequestValidationError

A key is not one of the options.

to_upload_method

to_upload_method(request: FileUpload) -> UploadFile

Build the method that stores a file.

Parameters:

Name Type Description Default
request FileUpload

The shared upload.

required

Returns:

Type Description
UploadFile

UploadFile with the file, its expiry and the options.

Raises:

Type Description
RequestValidationError

The echo API cannot take the upload, such as an expiry outside an hour to thirty days.

to_retrieve_method

to_retrieve_method(file_id: str) -> RetrieveFile

Build the method that reads a file's metadata.

Parameters:

Name Type Description Default
file_id str

The file's id.

required

Returns:

Type Description
RetrieveFile

RetrieveFile.

to_list_method

to_list_method(cursor: str | None) -> ListFiles

Build the method that reads one page of the stored files.

Parameters:

Name Type Description Default
cursor str | None

next of the page before, or None for the first page.

required

Returns:

Type Description
ListFiles

ListFiles of page_size files.

to_page

to_page(
    raw: EchoFilePage,
) -> tuple[Sequence[EchoFile], str | None]

Take the files of a page and where the next one starts.

Parameters:

Name Type Description Default
raw EchoFilePage

The page.

required

Returns:

Type Description
tuple[Sequence[EchoFile], str | None]

Its files and its next.

to_delete_method

to_delete_method(file_id: str) -> DeleteFile

Build the method that deletes a file.

Parameters:

Name Type Description Default
file_id str

The file's id.

required

Returns:

Type Description
DeleteFile

DeleteFile.

to_download_method

to_download_method(
    file: ProviderFile,
) -> DownloadFile | DownloadContainerFile

Build the method that streams a file's content.

Parameters:

Name Type Description Default
file ProviderFile

The file.

required

Returns:

Type Description
DownloadFile | DownloadContainerFile

DownloadContainerFile for a file of a container, else

DownloadFile | DownloadContainerFile

DownloadFile.

to_stored_file

to_stored_file(raw: EchoFile) -> StoredFile[EchoFile]

Convert the echo file into the shared metadata.

Parameters:

Name Type Description Default
raw EchoFile

The echo file.

required

Returns:

Type Description
StoredFile[EchoFile]

Its id, name, bytes as size and its times in UTC.

methods

Methods of the echo API: generation, files and container files.

Classes:

Name Description
GenerateParams

Fields of POST /echo, buffered or streamed.

Generate

Generate a response and wait for the whole of it.

StreamGenerate

Generate a response and stream its events.

UploadFile

Store a file; never retried, since it is not idempotent.

ListFiles

Read one page of the stored files, newest first.

RetrieveFile

Read a file's metadata.

DeleteFile

Delete a file.

DownloadContainerFile

Stream the content of a file in a code container.

DownloadFile

Stream a file's content.

Attributes:

Name Type Description
TERMINAL_EVENTS Final

The event that ends a complete stream.

MIN_EXPIRY Final

The shortest expires_in_seconds an upload takes: an hour.

MAX_EXPIRY Final

The longest expires_in_seconds an upload takes: thirty days.

TERMINAL_EVENTS module-attribute

TERMINAL_EVENTS: Final = EventTypes('done')

The event that ends a complete stream.

MIN_EXPIRY module-attribute

MIN_EXPIRY: Final = 3600

The shortest expires_in_seconds an upload takes: an hour.

MAX_EXPIRY module-attribute

MAX_EXPIRY: Final = 2592000

The longest expires_in_seconds an upload takes: thirty days.

GenerateParams

Bases: BaseAPIType

Fields of POST /echo, buffered or streamed.

Every field but model and input is left out until it is given.

Attributes:

Name Type Description
model Annotated[str, Field(min_length=1)]

Model identifier.

input Annotated[list[dict[str, Any]], Field(min_length=1)]

The conversation: messages, turns of calls and results.

instructions str | MISSING

A system prompt.

max_output_tokens Annotated[int, Field(gt=0)] | MISSING

Upper bound on generated tokens.

end_user_id Annotated[str, Field(max_length=64)] | MISSING

An opaque id of the end user.

effort str | MISSING

How much work the model puts in, such as high.

output_schema dict[str, Any] | MISSING

A JSON Schema the text must follow.

tools list[dict[str, Any]] | MISSING

The tools the model may call, and the built-in tools it may use, each an object with a type (web_search, code_execution, ...) in place of name.

tool_choice str | dict[str, Any] | MISSING

auto, none, required or {"name": ...} of one tool.

parallel_tool_calls bool | MISSING

Whether several calls may come at once.

temperature Annotated[float, Field(ge=0, le=2)] | MISSING

Sampling temperature, 0 to 2.

stop list[str] | MISSING

Texts that end the generation where they appear.

Generate

Bases: GenerateParams, BaseAPIMethod[EchoResponse]

Generate a response and wait for the whole of it.

A transient failure is retried.

Example

response = await client.send_request( ... Generate(model="echo-1", input=[{"role": "user", ...}]) ... )

StreamGenerate

Bases: GenerateParams, BaseStreamMethod[EchoEvent]

Generate a response and stream its events.

The stream is start, text deltas and tool calls (each a tool_call_start, tool_call_delta pieces and a tool_call_done), and done; an error event is raised as an APIStatusError. Only opening the stream is retried, and a body that ends before done raises APIStreamInterruptedError.

Attributes:

Name Type Description
stream Literal[True]

Always True: it asks for server-sent events.

Example

method = StreamGenerate(model="echo-1", input=[...]) async with client.stream(method) as events: ... async for event in events: ... ...

UploadFile

Bases: BaseAPIMethod[EchoFile]

Store a file; never retried, since it is not idempotent.

Attributes:

Name Type Description
file InputFile

The file.

purpose str | MISSING

What the file is for.

expires_in_seconds Annotated[int, Field(ge=MIN_EXPIRY, le=MAX_EXPIRY)] | MISSING

Seconds until the file expires, an hour to thirty days; kept without one.

Example

stored = await client.send_request(UploadFile(file=path))

ListFiles

Bases: BaseAPIMethod[EchoFilePage]

Read one page of the stored files, newest first.

Attributes:

Name Type Description
limit Annotated[int, Field(ge=1, le=1000)]

Files in the page.

after str | MISSING

next of the page before; the first page without it.

Example

page = await client.send_request(ListFiles(limit=100))

RetrieveFile

Bases: BaseAPIMethod[EchoFile]

Read a file's metadata.

Attributes:

Name Type Description
file_id str

The file's id.

Example

stored = await client.send_request(RetrieveFile(file_id="f1"))

DeleteFile

Bases: BaseAPIMethod[EchoDeleted]

Delete a file.

Attributes:

Name Type Description
file_id str

The file's id.

Example

await client.send_request(DeleteFile(file_id="f1"))

DownloadContainerFile

Bases: BaseStreamMethod[bytes]

Stream the content of a file in a code container.

Attributes:

Name Type Description
container_id str

The container's id.

file_id str

The file's id in it.

Example

method = DownloadContainerFile(container_id="c1", file_id="f1") async with client.stream(method) as content: ... await content.save(path)

DownloadFile

Bases: BaseStreamMethod[bytes]

Stream a file's content.

Attributes:

Name Type Description
file_id str

The file's id.

Example

async with client.stream(DownloadFile(file_id="f1")) as content: ... await content.save(path)

resolver

Error mapping of the echo API.

Classes:

Name Description
EchoErrorResolver

Error resolver of the echo API.

Attributes:

Name Type Description
QUOTA_CODE Final

error.code of an exhausted quota, over HTTP or in a stream.

ERROR_STATUSES Final[Mapping[str, int]]

HTTP status of each error code an error event of a stream carries.

QUOTA_CODE module-attribute

QUOTA_CODE: Final = 'insufficient_quota'

error.code of an exhausted quota, over HTTP or in a stream.

ERROR_STATUSES module-attribute

ERROR_STATUSES: Final[Mapping[str, int]] = {
    "invalid_request": HTTPStatus.BAD_REQUEST,
    QUOTA_CODE: HTTPStatus.TOO_MANY_REQUESTS,
    "overloaded": HTTPStatus.SERVICE_UNAVAILABLE,
}

HTTP status of each error code an error event of a stream carries.

EchoErrorResolver

Bases: BaseErrorResolver

Error resolver of the echo API.

Errors come in the common {"error": {"code", "message"}} shape, over HTTP and as the error event of a stream. An exhausted quota (insufficient_quota) is a QuotaExceededError and is never retried; everything else is classified by its status.

Methods:

Name Description
classify_status

Pick the error class and transience of the echo statuses.

classify_status

classify_status(
    *,
    status: int,
    code: str | None,
    error_type: str | None,
    message: str | None,
    retry_after: float | None,
) -> tuple[type[APIStatusError], bool]

Pick the error class and transience of the echo statuses.

Parameters:

Name Type Description Default
status int

HTTP status the error is classified by.

required
code str | None

Provider error code, if any.

required
error_type str | None

Provider error type, if any.

required
message str | None

Error message, if any.

required
retry_after float | None

Seconds from Retry-After, if any.

required

Returns:

Type Description
type[APIStatusError]

QuotaExceededError (never transient) for an exhausted

bool

quota, else the default classification.

types

Types of the echo API: its response, stream events, files and options.

Type Aliases:

Name Description
EchoEvent

An event of a stream; an unknown type is kept.

OutputItem

An item of a response's output; an unknown type is kept.

Classes:

Name Description
TextItem

Text the model wrote.

RefusalItem

The model's refusal to answer.

ToolCallItem

A call of one of the request's tools.

ServerToolItem

A call of a built-in tool, which the echo API ran itself.

CitationItem

A source the text rests on.

FileItem

A file a built-in tool made.

EchoUsage

Token counts of a generation.

EchoResponse

A finished generation, the answer of POST /echo.

StartEvent

The first event of a stream.

TextDeltaEvent

A piece of the text.

ToolCallStartEvent

The start of a tool call.

ToolCallDeltaEvent

A piece of a call's arguments.

ToolCallDoneEvent

The end of a tool call: its arguments are complete.

ServerToolStartEvent

A built-in tool call begins.

ServerToolDoneEvent

A built-in tool call has its result, or failed.

CitationEvent

A source of the text.

FileEvent

A file a built-in tool made.

DoneEvent

The last event of a complete stream.

EchoFile

A stored file.

EchoFilePage

A page of the stored files, newest first.

EchoDeleted

The answer to a deletion.

EchoGenerationOptions

Fields of POST /echo that the shared request does not hold.

EchoFileOptions

Fields of POST /files that the shared upload does not hold.

Attributes:

Name Type Description
OUTPUT_TAGS

Tag of each output type the echo types model.

EVENT_TAGS

Tag of each event type the echo types model.

OUTPUT_TAGS module-attribute

OUTPUT_TAGS = {
    "text": "text",
    "refusal": "refusal",
    "tool_call": "tool_call",
    "server_tool": "server_tool",
    "citation": "citation",
    "file": "file",
}

Tag of each output type the echo types model.

EVENT_TAGS module-attribute

EVENT_TAGS = {
    "start": "start",
    "text_delta": "text_delta",
    "tool_call_start": "tool_call_start",
    "tool_call_delta": "tool_call_delta",
    "tool_call_done": "tool_call_done",
    "server_tool_start": "server_tool_start",
    "server_tool_done": "server_tool_done",
    "citation": "citation",
    "file": "file",
    "done": "done",
}

Tag of each event type the echo types model.

OutputItem

OutputItem = Annotated[
    Annotated[TextItem, Tag("text")]
    | Annotated[RefusalItem, Tag("refusal")]
    | Annotated[ToolCallItem, Tag("tool_call")]
    | Annotated[ServerToolItem, Tag("server_tool")]
    | Annotated[CitationItem, Tag("citation")]
    | Annotated[FileItem, Tag("file")]
    | Annotated[BaseUnknown, Tag(UNKNOWN)],
    Discriminator(by_type(OUTPUT_TAGS)),
]

An item of a response's output; an unknown type is kept.

EchoEvent

EchoEvent = Annotated[
    Annotated[StartEvent, Tag("start")]
    | Annotated[TextDeltaEvent, Tag("text_delta")]
    | Annotated[ToolCallStartEvent, Tag("tool_call_start")]
    | Annotated[ToolCallDeltaEvent, Tag("tool_call_delta")]
    | Annotated[ToolCallDoneEvent, Tag("tool_call_done")]
    | Annotated[
        ServerToolStartEvent, Tag("server_tool_start")
    ]
    | Annotated[
        ServerToolDoneEvent, Tag("server_tool_done")
    ]
    | Annotated[CitationEvent, Tag("citation")]
    | Annotated[FileEvent, Tag("file")]
    | Annotated[DoneEvent, Tag("done")]
    | Annotated[BaseUnknown, Tag(UNKNOWN)],
    Discriminator(by_type(EVENT_TAGS)),
]

An event of a stream; an unknown type is kept.

TextItem

Bases: BaseAPITypeExtra

Text the model wrote.

Attributes:

Name Type Description
type Literal['text']

Always text.

text str

The text.

RefusalItem

Bases: BaseAPITypeExtra

The model's refusal to answer.

Attributes:

Name Type Description
type Literal['refusal']

Always refusal.

text str

What the model says about it.

ToolCallItem

Bases: BaseAPITypeExtra

A call of one of the request's tools.

Attributes:

Name Type Description
type Literal['tool_call']

Always tool_call.

id str

The call's id, which its result names.

name str

The tool.

arguments str

The arguments, as JSON text.

ServerToolItem

Bases: BaseAPITypeExtra

A call of a built-in tool, which the echo API ran itself.

Attributes:

Name Type Description
type Literal['server_tool']

Always server_tool.

id str

The call's id.

kind str

What it does: web_search, code_execution or another kind of ServerToolKind.

status str

in_progress, completed, incomplete or failed.

queries list[str]

What a search looked for.

code str | None

The code a code execution ran.

error str | None

What went wrong, for a failed call.

result dict[str, Any] | None

The call's result, in the echo API's own words.

CitationItem

Bases: BaseAPITypeExtra

A source the text rests on.

Attributes:

Name Type Description
type Literal['citation']

Always citation.

kind str

url, file or container_file.

url str | None

The page, for url.

title str | None

The title of the page or file.

file_id str | None

The file's id, for a file.

container_id str | None

The container's id, for container_file.

filename str | None

The file's name.

cited_text str | None

The cited words of the source.

start_index int | None

Where the cited part of the text starts.

end_index int | None

Where it ends.

FileItem

Bases: BaseAPITypeExtra

A file a built-in tool made.

Attributes:

Name Type Description
type Literal['file']

Always file.

id str

The file's id.

filename str | None

Its name.

container_id str | None

The container that holds it; None for a file kept with the stored files.

call_id str | None

The built-in tool call that made it.

EchoUsage

Bases: BaseAPITypeExtra

Token counts of a generation.

Attributes:

Name Type Description
input_tokens int

Tokens of the request.

output_tokens int

Tokens generated.

EchoResponse

Bases: BaseAPITypeExtra

A finished generation, the answer of POST /echo.

Attributes:

Name Type Description
id str

The generation's id.

model str

The model that generated it.

output list[OutputItem]

What the model wrote and called, in order.

stop str

Why it stopped: end or tool_calls when it reached its end, max_tokens, refusal, paused or another reason when it stopped early.

usage EchoUsage

Its token counts.

StartEvent

Bases: BaseAPITypeExtra

The first event of a stream.

Attributes:

Name Type Description
type Literal['start']

Always start.

id str

The generation's id.

model str

The model that generates it.

TextDeltaEvent

Bases: BaseAPITypeExtra

A piece of the text.

Attributes:

Name Type Description
type Literal['text_delta']

Always text_delta.

delta str

The piece.

ToolCallStartEvent

Bases: BaseAPITypeExtra

The start of a tool call.

Attributes:

Name Type Description
type Literal['tool_call_start']

Always tool_call_start.

id str

The call's id.

name str

The tool.

ToolCallDeltaEvent

Bases: BaseAPITypeExtra

A piece of a call's arguments.

Attributes:

Name Type Description
type Literal['tool_call_delta']

Always tool_call_delta.

id str

The call's id.

delta str

The piece of the JSON text.

ToolCallDoneEvent

Bases: BaseAPITypeExtra

The end of a tool call: its arguments are complete.

Attributes:

Name Type Description
type Literal['tool_call_done']

Always tool_call_done.

id str

The call's id.

ServerToolStartEvent

Bases: BaseAPITypeExtra

A built-in tool call begins.

Attributes:

Name Type Description
type Literal['server_tool_start']

Always server_tool_start.

call ServerToolItem

The call as it is known at the start.

ServerToolDoneEvent

Bases: BaseAPITypeExtra

A built-in tool call has its result, or failed.

Attributes:

Name Type Description
type Literal['server_tool_done']

Always server_tool_done.

call ServerToolItem

The whole call.

CitationEvent

Bases: BaseAPITypeExtra

A source of the text.

Attributes:

Name Type Description
type Literal['citation']

Always citation.

citation CitationItem

The source.

FileEvent

Bases: BaseAPITypeExtra

A file a built-in tool made.

Attributes:

Name Type Description
type Literal['file']

Always file.

file FileItem

The file.

DoneEvent

Bases: BaseAPITypeExtra

The last event of a complete stream.

Attributes:

Name Type Description
type Literal['done']

Always done.

stop str

Why the generation stopped, as EchoResponse.stop.

usage EchoUsage

Its token counts.

refusal str | None

What the model says about a refusal, if it refused.

EchoFile

Bases: BaseAPITypeExtra

A stored file.

Attributes:

Name Type Description
id str

The file's id.

filename str

Its name.

bytes int

Its size in bytes.

created_at int

When it was stored, in seconds since the epoch.

expires_at int | None

When it expires, in seconds since the epoch; None if it is kept.

EchoFilePage

Bases: BaseAPITypeExtra

A page of the stored files, newest first.

Attributes:

Name Type Description
data list[EchoFile]

The page's files.

next str | None

Where the next page starts; None after the last one.

EchoDeleted

Bases: BaseAPITypeExtra

The answer to a deletion.

Attributes:

Name Type Description
id str

The deleted file's id.

deleted bool

Always True.

EchoGenerationOptions

Bases: TypedDict

Fields of POST /echo that the shared request does not hold.

Given to client.generation.create() or stream() as provider_options.

Attributes:

Name Type Description
temperature Annotated[float, Field(ge=0, le=2)]

Sampling temperature, 0 to 2.

stop list[str]

Texts that end the generation where they appear.

EchoFileOptions

Bases: TypedDict

Fields of POST /files that the shared upload does not hold.

Given to client.files.upload() as provider_options.

Attributes:

Name Type Description
purpose str

What the file is for.