Tool calls¶
A tool call is a request from the model for your application to run a
function. The library sends tool definitions and returns calls; it does not
run functions unless you use generation.tool_runner. Either way, your
application decides how to handle failures, how many rounds to allow, and
whether a call is safe to run.
import json
from typing import Any
from aiollm import (
GenerationResponse,
IGenerationClient,
InputItem,
Tool,
ToolResult,
Turn,
tool,
)
@tool
def get_weather(city: str) -> dict[str, str | int]:
"""Current weather in a city."""
return {"city": city, "celsius": 15}
async def ask_with_tools(
client: IGenerationClient, question: str
) -> GenerationResponse[Any]:
conversation: list[InputItem] = [Turn.user(question)]
while True:
response = await client.generation.create(
model="gpt-6-astra",
input=conversation,
tools=get_weather,
)
if response.incomplete_reason is not None or not response.tool_calls:
return response
conversation.append(response.to_turn())
for call in response.tool_calls:
if call.name != "get_weather":
raise ValueError(f"Unknown tool: {call.name}")
args, kwargs = call.parse_arguments(get_weather)
result = get_weather(*args, **kwargs)
output = result if isinstance(result, str) else json.dumps(result)
conversation.append(ToolResult(call_id=call.id, output=output))
Define a callable tool¶
@tool wraps a function without changing how you call it in Python. Its
cached Tool definition is generated from the function name, parameter
annotations and docstring. Pass the callable itself as tools=get_weather or
include it in a sequence such as tools=[get_weather, get_time]; there is no
need to write get_weather.tool or construct a GenerationRequest.
The same keyword fields are accepted by generation.create() and
generation.stream(). They build a GenerationRequest internally. Pass a
ready GenerationRequest only when you already have one or want to reuse it;
do not pass both a request and request fields.
An undecorated callable is accepted too, but its schema is generated when the
request is built. Decorate reusable functions once to cache their schema.
Tool.from_callable() returns a callback-bearing Tool definition;
Tool.from_model() and Tool(...) define a schema without a callback.
Tool.from_model() is useful when you want to validate arguments as a
Pydantic model and dispatch the call yourself.
On the wire, a tool definition has a name, an optional description, a
parameters JSON Schema object, and an optional strict flag. Names may use
letters, digits, _ and -, are limited to 128 characters, and must be
unique within the request. The schema is sent as written; each provider may
support only part of JSON Schema.
Tool.from_callable(function, description=..., strict=...) uses the
function's name, parameter annotations and docstring; description=
overrides the docstring. @tool(description=..., strict=...) provides the
same options and exposes the cached definition as .tool. Neither executes
the function while defining it.
Any callable works. A functools.partial takes the name and docstring of the
function it wraps, and its bound arguments are left out of the schema, so
functools.partial(get_weather, api_key) keeps the key away from the model.
An object with a __call__ method takes the name of its class, the class
docstring (or else the method's), and the method's parameters without self.
*args becomes an array property and **kwargs an object property, each named
after its parameter; parse_arguments() puts them back in their places. The
schema has additionalProperties: false, since any other key is refused.
Manual buffered workflow¶
The first example sends a request with a callable tool. When the model returns
calls, response.to_turn() adds its turn to the conversation and each
ToolResult answers one call. The next create() continues that conversation.
The while loop is yours to bound; a production agent should set a maximum
number of rounds and decide what to do if a tool fails.
ToolCall.parse_arguments() without an argument parses JSON into an object
but does not validate it against the tool schema. Pass the callable or a
Pydantic model to validate: for a callable it returns (args, kwargs), and
for a model it returns the validated model. strict=True asks the provider to
enforce the schema too; local validation is still useful when strict mode is
off.
ToolResult.output is text. Serialize non-string results as JSON. To report a
tool failure, put an error description in the output; there is no separate
error flag. The adapters check that every call has exactly one result, in
order, before sending the next request.
response.to_turn() preserves the provider's native output alongside the
shared calls and text. This lets the same provider continue with its own
reasoning data; another provider receives only the shared conversation.
If incomplete_reason is set, do not run the calls: the last call may be
truncated and to_turn() refuses to continue that response.
Choose calls¶
tool_choice accepts ToolChoice.AUTO (the default), ToolChoice.NONE,
ToolChoice.REQUIRED, or ForcedTool(name=...) to require one named tool.
parallel_tool_calls=False limits the response to one call; both providers
allow several by default. These options require tools; a forced name not in
the request and parallel_tool_calls with ToolChoice.NONE are rejected
before sending.
With strict=True, the generated schema must have
additionalProperties: false and list every property in required;
otherwise the providers return 400. A parameter with a default, or *args,
is not required, so a strict callable has neither. A callable with
**kwargs cannot be strict at all, since its arguments are an object of any
keys: Tool.from_callable() and @tool raise TypeError. Without
strict=True, neither provider requests strict schema adherence.
Automatic callback runner¶
The runner is available on a concrete provider resource such as
OpenAIClient.generation.tool_runner. It does not belong to the generic
IGenerationResource protocol, which guarantees only create() and
stream(). The runner invokes registered callbacks and packages their
results, but it does not send the next model request; append the response turn
and results, then call generation again as in this example.
tool_runner.create() waits for the full response and returns an async
iterator. The callback execution starts when the iterator is entered
(async with calls:) or first read, whichever comes first, not when
create() returns. It binds and validates every call before running the
first callback; an unknown tool, invalid arguments or missing callback
therefore fails without running any callback. Set concurrency on
tool_runner.create() or tool_runner.stream() to limit how many callbacks
run at once; it defaults to 1, preserving sequential execution. Results
are yielded in tool-call order even when callbacks finish in a different
order, and each result keeps its matching call_id; pass in_order=False to
get each result as soon as its callback finishes. A finished result that has
not been read yet keeps its slot, so with tool-call order a slow first
callback can hold back the next ones: buffer_size (default 0) is how many
such results may wait beyond concurrency while further callbacks start; once
they are all taken, the next callback waits again. Entering the iterator or
reading it for the first time starts the first callbacks, up to
concurrency; after that, further callbacks are started only while the
consumer waits for a result, not between its reads. A streamed call starts its
callbacks once the stream has ended with its CompletedEvent and the consumer
reads past it.
The runner executes callbacks with
aiopulse's CallbackExecutor.
When a callback or the JSON serialization of its output fails, the runner
cancels unfinished callbacks, waits for them, and raises a
CallbackExceptionGroup (from aiollm import CallbackExceptionGroup) from
the iteration, even for a single failure. It holds every failure observed
before the callbacks stopped, so except* ValueError: handles them by type,
and its failures give each one's index, the position of its tool call. A
callback that ends with a cancellation the runner did not request, for example
by awaiting a cancelled task, is one of those failures, as a
CallbackCancelledError whose __cause__ is the original CancelledError;
only a cancellation of the consumer itself propagates as CancelledError,
after unfinished callbacks have been cancelled and awaited. Callbacks already
started may have performed side effects before cancellation.
To let the model read a failure instead, pass on_error: it is called with
the Exception of a callback, or of serializing its output, and what it
returns becomes that call's ToolResult.output (a string as is, anything else
as JSON), while the other callbacks go on. For example,
on_error=lambda error: f"Error: {error}" sends the exception's message; keep
in mind that the message goes to the provider. An exception raised by
on_error itself is the call's failure, and a cancellation is never passed to
it.
If buffered results are only partially consumed, use the iterator as an async
context manager (async with calls:) or call await calls.aclose() to cancel
and wait for unfinished callbacks; closing raises the failures recorded so far,
unless the block is already raising an exception, which then takes precedence.
An iteration still waiting in another task ends with StopAsyncIteration. The
streaming context closes its callbacks the same way when it exits.
Async callbacks (async def, or an object whose __call__ is one) run in the
event loop. Any other callback runs in a worker thread (asyncio.to_thread),
so a blocking one does not stop the loop, and with concurrency above 1
synchronous callbacks run at the same time: they must then be thread-safe, and
an object bound to the thread that made it (an sqlite3 connection, for
example) must be used from an async def callback instead. A thread cannot be
stopped: a cancelled synchronous callback runs to its end after the runner has
closed, and a failure it then ends with is reported through the event loop's
exception handler, even after the coroutine given to asyncio.run has
returned, since the loop waits for its worker threads before it closes.
The response is available as .original. An incomplete response has no
results and does not invoke callbacks; inspect incomplete_reason before
trying to continue it. Strings returned by callbacks become ToolResult.output
as-is; other values are encoded with json.dumps() and must be JSON
serializable.
from typing import Any
from aiollm import GenerationResponse, InputItem, Turn, tool
from aiollm_openai import OpenAIClient
@tool
def get_weather(city: str) -> dict[str, str | int]:
"""Get the current weather in a city."""
return {"city": city, "celsius": 15}
async def ask_with_runner(
client: OpenAIClient, question: str
) -> GenerationResponse[Any]:
conversation: list[InputItem] = [Turn.user(question)]
while True:
calls = await client.generation.tool_runner.create(
model="gpt-6-astra",
input=conversation,
tools=get_weather,
concurrency=4,
)
response = calls.original
if response.incomplete_reason is not None or not response.tool_calls:
return response
conversation.append(response.to_turn())
async with calls:
conversation.extend([result async for result in calls])
tool_runner.stream() is an async context manager. It yields the generation's
events, then yields ToolResults only after a successful CompletedEvent.
Consume it to exhaustion to receive the results. If the stream fails or ends
early, callbacks are not run; an incomplete CompletedEvent is yielded but
produces no results. As with create(), the caller decides whether to continue
the conversation. Only append completed.to_turn() when the completion is
not incomplete; a truncated tool turn cannot be continued.
This example runs one streamed round and returns its completion and results.
The caller can append completed.to_turn() followed by the results before
the next generation call.
from typing import Any
from aiollm import CompletedEvent, InputItem, ToolResult, tool
from aiollm_openai import OpenAIClient
@tool
def get_weather(city: str) -> dict[str, str | int]:
"""Get the current weather in a city."""
return {"city": city, "celsius": 15}
@tool
def get_time() -> str:
"""Get the current time."""
return "3 PM"
async def stream_round(
client: OpenAIClient, conversation: list[InputItem]
) -> tuple[CompletedEvent[Any], list[ToolResult]]:
completed: CompletedEvent[Any] | None = None
results: list[ToolResult] = []
async with client.generation.tool_runner.stream(
model="gpt-6-astra",
input=conversation,
tools=[get_weather, get_time],
concurrency=2,
) as events:
async for event in events:
if isinstance(event, CompletedEvent):
completed = event
elif isinstance(event, ToolResult):
results.append(event)
if completed is None:
raise RuntimeError("The stream ended without a completed event")
return completed, results
A tool can take its arguments from a Pydantic model: Tool.from_model(name, Model, ...) sends the model's JSON Schema as parameters, and parse_arguments(Model) gives the arguments back as the model. For strict=True, the model forbids extra fields and has no defaults, since the providers want additionalProperties: false and every property required:
from pydantic import BaseModel, ConfigDict, ValidationError
class City(BaseModel):
model_config = ConfigDict(extra="forbid")
city: str
CITY_WEATHER = Tool.from_model(
"get_weather",
City,
description="Current weather in a city.",
strict=True,
)
def answer(call: ToolCall) -> ToolResult:
try:
arguments = call.parse_arguments(City)
except ValidationError as error:
output = json.dumps({"error": str(error)})
else:
output = json.dumps(get_weather(arguments.city))
return ToolResult(call_id=call.id, output=output)
A streamed call can start a tool before the response ends. Start that way only the tools that are safe to run for a generation that then fails, and run the others once the stream has ended with its CompletedEvent:
generation.stream() emits ToolCallStartEvent, argument
ToolCallDeltaEvents and ToolCallDoneEvent as the model writes a call.
ToolCallDoneEvent means that call's arguments are complete, not that the
generation succeeded: the stream can still fail before CompletedEvent.
events.final() returns the terminal event after iteration and preserves the
provider output for completed.to_turn().
The example starts only a read-only callback early. Cancelling its asyncio task
does not stop work already running in asyncio.to_thread, so work started
early must remain safe even if the generation later fails.
import asyncio
from aiollm import ToolCall, ToolCallDoneEvent
READ_ONLY = {"get_weather"}
def run(call: ToolCall) -> str:
if call.name == "get_weather":
args, kwargs = call.parse_arguments(get_weather)
result = get_weather(*args, **kwargs)
return result if isinstance(result, str) else json.dumps(result)
return json.dumps({"error": f"unknown tool {call.name}"})
async def stream_round(
client: IGenerationClient, conversation: list[InputItem]
) -> list[InputItem]:
early: dict[str, asyncio.Task[str]] = {}
try:
async with client.generation.stream(
model="gpt-6-astra", input=conversation, tools=get_weather
) as events:
async for event in events:
if (
isinstance(event, ToolCallDoneEvent)
and event.call.name in READ_ONLY
):
early[event.call.id] = asyncio.create_task(
asyncio.to_thread(run, event.call)
)
completed = await events.final()
except BaseException:
for task in early.values():
task.cancel()
raise
# The generation succeeded: now the tools with side effects run.
results: list[InputItem] = [completed.to_turn()]
for call in completed.tool_calls:
output = await early[call.id] if call.id in early else run(call)
results.append(ToolResult(call_id=call.id, output=output))
return [*conversation, *results]
Built-in tools¶
The shared callable workflow covers function tools defined by your
application. Tools the provider runs itself (web search, code execution, file
search, MCP, ...) are built-in tools: WebSearch(), CodeExecution() and
ProviderTool(...) go in the same tools= list, their calls come back in
response.server_tool_calls and never in tool_calls, and nothing runs in
your code; see Built-in tools. tool_runner executes
callbacks attached to shared Tool definitions only; a built-in tool needs no
callback.
What each field becomes for each provider is listed in Shared field mapping.