Errors and retries¶
AIAPIError
├── APIStatusError the API reported a failure
│ ├── BadRequestError 400
│ ├── AuthenticationError 401
│ ├── BillingError 402
│ ├── PermissionDeniedError 403
│ ├── NotFoundError 404
│ ├── RequestTimeoutError 408
│ ├── ConflictError 409
│ ├── ContentTooLargeError 413
│ ├── UnprocessableEntityError 422
│ ├── RateLimitError 429
│ ├── QuotaExceededError no status of its own: a quota error
│ ├── InternalServerError 500
│ ├── BadGatewayError 502
│ ├── ServiceUnavailableError 503
│ └── GatewayTimeoutError 504
├── APITransportError no valid HTTP response
│ ├── APIConnectionError also a body cut off by a lost connection
│ ├── APITimeoutError
│ ├── APITLSError
│ ├── APIRedirectError redirect not followed
│ ├── APIRequestError failed upload source, invalid header or URL
│ └── APIStreamInterruptedError an open stream broke or ended early
└── APIResponseError invalid HTTP, body too large, not decodable or valid
The three branches answer the question "what came back?":
APIStatusError: the API reported a failure. Usually that is an error status (400 or more); a status without its own class is raised asAPIStatusErroritself. It is also a failure reported in-band after a successful status, such as a stream's error event or a result that says it failed; itsstatusis then that successful one (see Error resolvers).APITransportError: no usable answer came back, because the request failed on the way.APIResponseError: an answer came back, but it cannot be used. It is not valid HTTP, it is larger thanmax_body_size, or it does not decode or validate.
An APIStatusError carries:
status,code,error_typeandrequest_id;provider, from the error resolver;retry_after, from theRetry-Afterheader;transient;raw, the parsed object that reported the failure, when there is one: OpenAI'sResponsewithstatus: "failed", or a stream's error event when it fits the item type.
A status error's message starts with the class's fixed description and ends with detail: <the API's message> when the API sent one, so logs have a stable prefix and the provider's own explanation. Transport and response errors carry their own message.
One more library error is raised before anything is sent: RequestValidationError (a ValueError), for a shared request that a provider cannot take; see Shared generation.
from aiollm import APIStatusError, APITransportError, RateLimitError
async def ask(prompt: str) -> str | None:
method = CreateCompletion(
model="example-1",
messages=[Message(role="user", content=prompt)],
)
try:
completion = await session.send_request(method)
except RateLimitError as error: # retries did not help
print(f"Rate limited, request {error.request_id}")
return None
except APIStatusError as error:
print(f"API error {error.status}, code {error.code}")
raise
except APITransportError as error:
print(f"Network error, transient: {error.transient}")
raise
return completion.text
When a request is retried¶
A request is retried only when the method sets __retryable__ = True, does not carry a one-shot file, and the error's transient flag is true. By default that flag is true for:
- statuses 408, 500, 502, 503, 504, and a provider's own transient statuses, which its resolver classifies (Anthropic's 529 "overloaded");
- status 429 when the response has
Retry-After; - connection errors, including a body cut off before its end, and timeouts.
It is false for certificate errors, redirects that are not followed, invalid requests and failing upload sources, since repeating them gives the same result, and APIResponseError is never retried.
How often and how long to retry is set by a RetryPolicy, the retry_policy argument of a session or a client (below). The default one makes at most four attempts, waits with random exponential backoff of up to 30 s but never less than Retry-After, raises the error at once when the API asks to wait longer than 60 s, and starts no retry 15 minutes after the first attempt; a wait that would end later is not started, and an attempt in progress is limited by its own timeout.
The reasons behind these rules:
- Retries are opt-in per method. Most AI endpoints are
POSTs that are not idempotent. A request that timed out may still have run on the server, and sending it again can start a second billed generation. The author of a method knows whether that is acceptable; the library does not. The library's provider methods opt in, as the official SDKs do, and accept the same risk. - A 429 is retried only with
Retry-After. Without it there is no telling a short rate limit from an exhausted quota, and retrying a quota error only delays the failure. With it, the server has said when to come back. Retry-Afteris capped. A server that asks for ten minutes would otherwise block the caller silently. Raising the error lets the caller decide, for example to queue the work.- The deadline bounds the total time. Four attempts with long waits and long timeouts could otherwise take far longer than any caller expects.
- Jitter spreads the retries. Many clients failing at the same moment don't all come back at the same moment.
- A policy narrows, never widens. Whether an error can succeed on a second try is the library's knowledge (
transient), and whether a request may run twice is the method's (__retryable__); a policy only tunes how far to go within both.
Retry policy¶
| Field | Default | Meaning |
|---|---|---|
max_attempts |
4 |
Attempts in total, the first one included; 1 turns retries off |
deadline |
900 |
Seconds after the first attempt after which no retry starts; None for no limit |
max_retry_after |
60 |
Longest Retry-After worth waiting for, in seconds |
backoff |
FullJitterBackoff() |
Gives the wait before each retry, from a RetryState; the session still waits at least Retry-After. Not called after the last attempt max_attempts allows; with a deadline, called for a wait the deadline then cuts, since the deadline depends on it |
retry_if |
None |
Asked about each transient error that max_attempts allows to retry, never the last attempt's; False raises it at once |
A RetryState holds the failed attempt's number (attempt, from 1), the seconds since the first attempt (elapsed) and its error, always a transient APIStatusError or APITransportError. FullJitterBackoff(multiplier=0.5, max_delay=30) draws the wait after attempt n from zero to multiplier * 2 ** (n - 1), capped at max_delay.
A policy given to send_request or stream replaces the session's for that call (for a stream, its opening):
from aiollm import RateLimitError, RetryPolicy, RetryState
def not_rate_limited(state: RetryState) -> bool:
return not isinstance(state.error, RateLimitError) # a 429 goes to the caller
patient = RetryPolicy(max_attempts=6, deadline=None, retry_if=not_rate_limited)
async def complete_patiently(prompt: str) -> Completion:
return await session.send_request(
CreateCompletion(
model="example-1",
messages=[Message(role="user", content=prompt)],
),
retry_policy=patient, # this call only; the session keeps its own
)
A policy applies only to methods with __retryable__ = True and to errors whose transient flag is true, and retry_if can refuse such an error but cannot make any other one retried. If retry_if or backoff raises, or backoff returns something other than a finite, non-negative number, that error is raised with the request's error as its __cause__. The settings are checked when the policy is created: a wrong type raises TypeError, a value out of range ValueError.
Error resolvers¶
Error responses are mapped by an error resolver. BaseErrorResolver reads:
- the common
{"error": {"code", "type", "message"}}body shape, and a stringerror({"error": "model not found"}, as Ollama sends) as the message; - the request id from the
X-Request-IdorRequest-Idheader, or from arequest_idfield of the body; Retry-After, in the two forms of RFC 9110: whole seconds in ASCII digits or an HTTP date. Any other form (1.5,-5,1e3) counts as no header, so a negative number cannot mean an immediate retry.
A body that is not JSON gives an error without a code or message, so an HTML error page from a proxy still becomes the right APIStatusError for its status.
A failure the API reports after a successful status (a stream's error event, a result that says it failed) is in-band. It is classified by the status that the resolver's error_statuses table gives its code, else its type, and it is never transient, whatever its class: the generation ran and failed, and running it again is your choice, as in the official SDKs, which retry by the HTTP status only.
Subclass the resolver to handle a provider's specifics: set provider and error_statuses, and override get_error_code or classify_status, which picks the class and transience of a status (the in-band rule stays the base class's):
from aiollm import APIStatusError, Provider, QuotaExceededError
class ExampleErrorResolver(BaseErrorResolver):
provider = Provider("example") # the provider's name
error_statuses = {"insufficient_quota": 429} # in a stream, too
def classify_status(
self,
*,
status: int,
code: str | None,
error_type: str | None,
message: str | None,
retry_after: float | None,
) -> tuple[type[APIStatusError], bool]:
if status == 429 and code == "insufficient_quota":
return QuotaExceededError, False # not worth retrying
return super().classify_status(
status=status,
code=code,
error_type=error_type,
message=message,
retry_after=retry_after,
)
The resolver returns the error instead of raising it, so the session raises it where the traceback is most useful. Error events inside streams go through the same resolver, so one classify_status covers both.
Library errors can be pickled and copied, for example to pass them between processes. They are rebuilt from their attributes without calling __init__ again, since __init__ composes the message and would add the prefix twice.