Skip to content

Client Integration

This page is for the people who put the toolkit inside an application — a web backend, a worker fleet, a batch pipeline — rather than for the people writing a single agent run. It is about the things an application has to own: which release it is running, when its processes have to restart, which models it offers, what it reconciles against an invoice, and which failures it must expect to handle itself.

Everything here links out to the page that owns the detail. Nothing is restated twice.

Pin a release, and know how to leave it

The toolkit is not on PyPI, so a Git ref is the version. Pin an immutable one — an annotated tag or a full 40-character SHA — install your requirements and the toolkit in one pip invocation, and keep the release's tested_constraints.txt if you want the exact set a release was verified against. A rollback is a re-install at the previous tag. All of that, with the commands, is in Quickstart.

Two habits are worth building into the deploy rather than the runbook:

  • Record the ref the image contains, and assert it at start-up from importlib.metadata.version("doc_processing_toolkit"). Without that, "we are on release X" is unverifiable at the moment you most need it to be true.
  • Diff the registry against your own model list before you ship. list_available_models() on the upgraded toolkit is the catalogue; a model your application offers that the registry no longer carries is rejected locally with a ValueError, which is the cheap failure, but only if you find it before a user does.

Restart your workers after an upgrade

Two pieces of state are resolved once per process and never re-read:

  • The model registry, loaded at first use. A long-lived worker keeps the catalogue it loaded at import, so upgrading the package on disk changes nothing until the process restarts. Every model addition, price change and capability flag is a registry edit, which means every one of them is invisible to a worker you did not restart.
  • .env, read at the first client initialization. Rotating an API key has no effect on a running process; pass api_key= explicitly if you need a different key without a restart. See Configuration.

So the upgrade sequence for a fleet is: install the new ref → restart every long-lived Python and MCP worker → refresh discovery → run a bounded smoke test through the same path the application uses.

Discovering models

Discovery is two MCP tools on the agent-factory server, list_available_models and get_model_info, or load_model_information() in Python. Both report the model's kind, not just its name: session_kinds says which session families it serves, and a model can serve several, because whisper-1 does transcription and translation and gpt-realtime does realtime and transcription on one socket. Filter the listing with modality="stt" or session_kind="realtime" rather than fetching every row and sorting it out in your own code — and note that realtime.open_realtime_session now uses the same field, so a kind a model does not serve is refused before a socket opens rather than after.

One asymmetry to code around: a conversational realtime row reports transcription because its own session transcribes what it hears, but it cannot be opened as a dedicated captions session. A dedicated kind is openable only on a row that declares it, so a "pick any model that lists this kind" router will eventually pick one that refuses. Keep the declared rows in your captions, translation and GPT-Live lists, and use input_transcription=True when what you want is a conversation's own transcript. See Which kind of session.

Read these fields with .get(). A field appears only when the registry knows the answer, so an absent supports_diarization means "the registry says nothing", which is not the same as false — whisper-1 declares supports_streaming_transcription: false on purpose, because the API accepts the flag and ignores it.

A model being retired carries a deprecation object: {"date", "replacement", "note"}, where a null date means a replacement is recommended without a shutdown having been announced. Poll that field rather than a changelog. Six rows carry one today: gpt-6-sol (it names gpt-6.1-sol, with no shutdown date), the two preview TTS rows, and three realtime rows, one of them a preview that names no replacement at all, which is how a preview row says so.

Populate separate model choices per task

Discovery returns every row — text, image, speech, realtime. Do not send every discovered ID through Agent; keep one list per task (text agents, images, recorded transcription, conversational voice) and check the chosen model's reasoning levels, input types and tool combinations before you offer it. get_model_info answers all of those.

Tiers, and what you can prove about a bill

service_tier is a request for the premium ("priority"), cheaper ("flex") or, on gpt-6-astra, much faster and dearer ("ultrafast") lane, and all are registry-gated: a model is offered a tier only if its row publishes a price for it. The important part for an integration is that billing follows the tier the provider actually served, not the one you asked for:

  • served_tier on each per-request record is what you were served.
  • tier_downgraded marks "you were not served the tier you asked for" — in either direction, so a flex request served at standard costs more than asked, not less.
  • served_tier = "unverifiable" means the provider sent no readable signal. Such a request is billed at standard, and only the provider's invoice settles it. On Google this is the normal outcome today, because every request streams and the streamed body carries no served-tier field the SDK surfaces.

The full rules, including the run-level service_tier_served and priority_request_count, are in Service Tiers.

Cost is a sum of per-request records

A run's cost is the sum of its per-request cost records, never one pricing pass over summed token totals — tiered models over-bill when you sum first. Reconcile from request_cost_records, which carries the model, the tokens, the tier and the cost of each individual API request. Since 0.5.1 cost also includes what the run's own tools reported spending (the multimedia MCP tools, the agent factory's create_agent and suggest_agent_config), in tool_reported_cost and tool_reported_costs; if your app added those costs itself, it now counts them twice (see Tool-reported spend).

cost_estimated on a record or a result means a token or tool value was estimated, not that the tier was uncertain. It is set when the provider reported no usage object, when a duration had to come from the byte heuristic, or when a model reports no output token count and the toolkit had to derive one — as gemini-3.5-transcribe does. Tier uncertainty has its own markers, above.

See Cost Tracking for the record schema and the cost-report recipe for a worked reconciliation.

Sessions are your state, and they are portable across processes, models and providers

An AgentSession serialises to a plain dict with to_dict() and restores with from_dict(), or to JSON text or an atomically written file with to_json() / from_json(), so a conversation can live in your database or on disk between HTTP requests. Checkpoint after every completed turn; the cadence, what a failed turn leaves behind and the resume rules are on History and Checkpointing, with the two store setups (none, a directory, an object store) under Setting up a store and the best practices for checkpointing and resuming. What an integration should hold onto:

  • provider and model on the envelope (version 3): who saved it. A restore under another model of the same provider is quiet; a restore of a provider-native history under another provider converts it (tool rounds and attachments cross; thinking, reasoning items and provider-run tool traffic are dropped and listed on conversion_report), and only a history that cannot be converted raises SessionCompatibilityError instead of failing with a 400 on the first request. session.rebind(new_agent) does the same in place between turns.

  • schema_version on the envelope. A reader accepts every version it knows: a missing version is the implicit version 1 and is migrated; a newer version loads with a warning rather than being refused, because refusing would strand a conversation the reader can still read.

  • The prefix fingerprint. A session records a digest of the system prompt and tools array it was built with. Resume it against a rebuilt prompt or a changed tool list and you get prefix_changed_on_resume plus a prefix_change_report — a report, never a refusal. It is the signal that this conversation's prompt cache, and any retained reasoning, restarts here.
  • Run boundaries. session.run_boundaries gives one {"run", "start", "end"} record per completed turn, so you can address "turn 3" without parsing provider-native messages.
  • truncate_to / truncate_to_turn implement "edit and resend" or "delete from turn k". The cut is a pure prefix truncation — nothing retained is rewritten, so signed reasoning blocks stay valid — and the cut point moves backward rather than stranding a tool call without its result. The cumulative counters are not rewound: they are the history of what the session spent.

Sessions has all of it, including what compaction does to the boundaries.

Late injections are the host's decision

A hook message that arrives after the last turn boundary of a run cannot be shown to that run. It is never dropped: the run returns it on AgentResult.undelivered_injections, and a session parks it in metadata["pending_injections"], which survives save and restore. With the default late_injection_policy="surface" your application decides what to do with it — normally, send it as the next user message. With "new_turn" the session sends that follow-up itself, bounded by max_follow_up_turns. See Sessions and Hooks.

A large tool surface does not have to sit in the prompt

If your agent exposes a big MCP tool surface, Agent(defer_tools=...) together with built_in_tools={"tool_search": BuiltInToolConfig(enabled=True)} moves most of that surface out of the cached prompt prefix on Anthropic and OpenAI. It is safe to set unconditionally: a model or provider without native tool search loads every tool eagerly and logs one warning rather than failing, so the same configuration survives a fallback chain that crosses providers. Read AgentResult.deferred_tools to see whether it took effect — an empty list beside a non-empty defer_tools means "loaded eagerly", and the reason is in the run's warnings. See Tool search and deferred tools.

To keep tools out altogether, give the server config an allowed_tools list. Since 0.5.1 every MCP format takes it: the toolkit filters the servers it runs itself (script, command, pre-built session, mcp_remote_client, and an in-process FastMCP server through {"type": "fastmcp", "server": ..., "allowed_tools": [...]}), OpenAI and Anthropic filter their provider-run servers, and Fireworks, which ignores the field, refuses it at construction. A typical use is an app that speaks through its own tool and wants only the multimedia server's image tools. See Choosing which tools a server exposes.

mcp_remote_client filtering was not reaching the model before 0.5.1

Its allowed_tools used to trim only the toolkit's own tool list; each provider lists the session's tools itself, so the model still saw every tool. The filter now sits on the session. An app that set the key was relying on behaviour it did not get; nothing else changes.

What hooks cannot do

A hook steers the loop; it does not interrupt it. Nothing in the API pre-empts a tool that is already executing. pause() and abort() are honoured at the next loop boundary: the in-flight tool runs to completion, its result is appended — every provider requires a result for every tool call it emitted — and only then does the run stop. skip_tool is the only veto, and it is offered at before_tool_call, before the tool starts.

An application that needs a hard stop in the middle of a long tool has to build it into the tool: a timeout, a cancellation token, a bounded subprocess. See Hooks and Steering.

A hook also sees only a preview of each tool result: the first 500 characters of its str(), as the stream does, with result_truncated saying whether that is all of it. An app that needs the whole result (a file path deep in a long JSON reply, say) sets Agent(tool_result_preview_chars=0), and the events then carry the whole result plus result_data, the result as JSON data. That setting never changes what the model is sent. See Tool results in events.

Realtime is a Python session API

open_realtime_session holds a provider WebSocket open from your backend. It is not an MCP tool and it is not a browser API. Your application owns audio capture, resampling (OpenAI wants 24 kHz in, Gemini Live 16 kHz), playback buffering, how much output was actually played, and binding the session's lifetime to a user disconnect or an idle timeout.

The connection between a browser and your own backend is a separate integration that the toolkit does not model. If you connect a browser directly to a provider instead, use that provider's temporary credential flow — never ship a permanent API key to a client.

Two behaviours to design around rather than discover: a turn boundary is not always the same event as "the model is idle" (see Knowing when a turn is over), and interrupting is two calls, not one — stop the response, then tell the provider how much of it was actually heard.

The session families do not share a shutdown story

A voice conversation is one of four session kinds, and the other three end differently enough that an integration should decide per family, not once:

  • Translation sessions end when your application says so. Neither vendor has a reliable end-of-stream event, so response_complete is opportunistic and close() is what flushes and drains the tail. Two consequences for a backend that streams a recording rather than a microphone: keep sending audio (including a second or two of silence at the end) or you lose the transcript tail, and allow for a close that is not instant — OpenAI answered session.close 3.9 s later in a live run, behind the audio it was still flushing, which is why that drain has its own 10-second budget (provider_options={"close_drain_seconds": ...} if your shutdown path needs a different ceiling).
  • A GPT-Live session has no end-of-response event at all. Nothing marks the end of a spoken answer, so drive your UI from playback and from the transcript, and bind the session's lifetime to your own idle timeout. Its bill is wall clock, not tokens — silence and backend thinking time included — so budget it by session length, and read session.cost_excludes before treating session.cost as the whole charge: a delegated Responses backend is billed separately, and the responses that ran are listed in session.delegated_usage for you to price.
  • Preview and entitlement are normal outcomes. gemini-3.5-live-translate-preview is a preview row; a refusal at connect for an account without access is a state to handle, not a bug, and the cheapest probe is a one-second session.

See Live translation and GPT-Live.

Kill switches and environment variables

Behaviour changes that could affect a running deployment ship with a switch, so you can turn one off without downgrading. They are all documented, with defaults, accepted values and when to reach for them, on one page: Configuration and Environment Variables. Treat that page as the authoritative list — it is not duplicated here, and a switch that exists only in a release note does not exist.

The shape to expect: a kill switch restores the previous behaviour, never a third one, and an unrecognised value is ignored with a warning rather than failing a process at start-up.

0.5.1 adds five, each on by default:

A few 0.5.1 changes ship without a switch, because the old behaviour was of no use or exposed what the caller meant to hide, and each is named in the CHANGELOG:

  • The multimedia MCP tools never return base64: the result carries the file's path. A client that read audio_base64 / images_base64 reads the file instead, or calls the multimedia.* Python functions, which return bytes. In a split deployment point DPT_MULTIMEDIA_OUTPUT_DIR at a shared volume. See Files, not base64.
  • allowed_tools now filters on every MCP format, including the provider-run Anthropic connector and mcp_remote_client, and is refused on Fireworks' provider-run connector. On the formats the toolkit runs it must be a list of names, checked at Agent(...): a bare string raises TypeError (on mcp_remote_client it used to be read as a set of characters, which hid every tool). A provider-run server's value is checked at Agent(...) too (on OpenAI a list or OpenAI's filter object), and OpenAI now receives an empty list, which exposes no tool, where the request used to drop it and expose every tool. A direct provider call (no Agent) is covered as well: Anthropic applies the key and Fireworks refuses it, where both used to ignore it.
  • An in-process FastMCP tool that returns an image (std-tools read_image, renderers, screenshots — see its with-the-toolkit and security guides) now delivers the image to the model instead of the text <...Image object at 0x...>, at image-token cost on the rows that accept images in tool results. A tool that returns an MCP TextContent, ImageContent or AudioContent object is delivered as that content too, where it used to arrive as the object's repr.
  • iteration_complete events cover every request of the tool loop since the previous event, so summing them gives the run's request cost up to the last event. The first event of a tool run now also carries the request that asked for the tools, so its cost is higher than before and its input_tokens counts two prompts: it is billed input, not how full the context is. See Hooks.
  • Pre-call compaction (summarize, combined) keeps the question the run is about to answer out of its summary, so the model answers it instead of replying that it is "already in the summary above". See Compaction.
  • A Gemini tool run that answers on its last allowed round succeeds (it used to report a max-iterations error beside the answer), and a Fireworks structured or plain-text run with a system prompt sends its schema instruction.

To let an agent in your app speak, draw and look at images without base64 reaching the model, see Speech and Images from an Agent in Your App.

Optional extras

The toolkit's install is deliberately lean, and an optional capability is an extra rather than a dependency everyone pays for. Today there is one:

Requires repository access (see Quickstart).

pip install "doc_processing_toolkit[audio] @ git+https://github.com/rzoeb/doc-processing-toolkit.git@<ref>"

[audio] installs a bundled ffmpeg so the toolkit can transcode a recording whose container the target speech-to-text model does not accept. Without it, that path raises a named error listing the accepted containers and this extra, rather than reaching the provider as a 400 — and a system ffmpeg, PyAV or DPT_FFMPEG_PATH satisfies it just as well. See Multimedia.