Architecture

The toolkit separates high-level agent orchestration from provider-specific implementations.

Main layers

1. Agent layer (doc_processing_toolkit.agent.*)
   • Entrypoint: Agent (direct constructor) and AgentBuilder (fluent builder pattern)
   • Tool loading, media transformation, provider routing, session support, cost aggregation
   • session, media_policy and history_conversion: the session envelope (to_json / from_json), what a save does with media (inline, a MediaStore, or stripped), and the neutral representation that carries a stored history across providers
2. Provider layer (doc_processing_toolkit.providers.*)
   • Provider-specific API calls for function calling and structured output
3. MCP server layer (doc_processing_toolkit.mcp_servers.*)
   • Three FastMCP servers: agent_factory (meta-agent workflows and A2A sessions), docs_server (documentation search/retrieval), and multimedia_server (TTS/STT/image tools, which write their output to files and return paths, never base64)
4. Media layer (doc_processing_toolkit.multimedia, doc_processing_toolkit.realtime)
   • multimedia: one-shot speech synthesis, transcription, and image generation/editing, plus speech streamed as it is generated (synthesize_speech_stream / asynthesize_speech_stream)
   • image_session: serialisable multi-turn image generation and editing on OpenAI and Google
   • audio_convert: optional speech-to-text container transcoding, with a discovered (never required) ffmpeg
   • realtime: provider-neutral speech-to-speech sessions over OpenAI Realtime and Gemini Live
   • realtime_translation, realtime_gpt_live: two further session families that register their own adapters into the same (provider, session_kind) routing table
   • Both sit beside the agent layer rather than inside it — they call providers directly and are not part of the tool-calling loop
5. Core utilities (doc_processing_toolkit.utils, data/model_information.json)
   • Client initialization, model metadata, token/cost helpers, and the one retry layer (tenacity over SDK clients built with max_retries=0)

Runtime flow (high level)

1. Build Agent with model, instructions, and optional tools.
2. Detect provider from model metadata.
3. Initialize provider client (initialize_clients or initialize_async_clients).
4. Route to function-calling or structured-output execution path.
5. Convert provider response to AgentResult.

Figure: agent lifecycle and session manager.

Figure: provider router decision flow.