Skip to content

Recipe: Speech and Images from an Agent in Your App

Goal

Let an agent that runs inside your application speak and make images, and get the audio and the images to your user: play a clip, show a picture. There are two patterns. Pick the first when your app embeds Agent in Python, and the second when the multimedia MCP server is already how your agents reach speech and images, or several apps share one server.

Why base64 must not reach the model

A tool's result goes back to the model, not to your app. A model can neither hear audio nor see an image that arrives as base64 text inside a tool result: to the model it is expensive text. A 10-second Google WAV is 480,000 bytes, which is 640,000 base64 characters, sent as input on that request and on every later one in the run; a 1024×1024 PNG is about 2 million. So in both patterns below the bytes go to your app and the model gets a short text result. Since 0.5.1 the multimedia MCP tools never return base64 at all (see Files, not base64).

Write a @tool that calls the multimedia function, hands the bytes to your player or UI, and returns a few words to the model. Report the call's cost with tool_spend_marker, so the agent's cost includes it (see Tool-reported spend).

from doc_processing_toolkit import Agent, tool
from doc_processing_toolkit.multimedia import synthesize_speech
from doc_processing_toolkit.utils import tool_spend_marker

PLAYER = ...  # your app's audio sink


@tool
def speak(text: str) -> dict:
    """Say the text aloud to the user."""
    clip = synthesize_speech("gpt-4o-mini-tts", text, voice="coral", output_format="mp3")
    spend = tool_spend_marker(clip["cost"], clip["cost_estimated"])
    if clip["error"]:
        return {"spoken": False, "error": clip["error"], **spend}  # a blocked request is still billed
    PLAYER.play(clip["audio_bytes"], clip["mime_type"])      # the bytes go to your app, not the model
    return {"spoken": True, "characters": len(text), **spend}


agent = Agent(model="claude-sonnet-5", instructions="Answer, then read the answer aloud.", tools=[speak])
result = agent.run_sync("What is the capital of Australia?")
result.cost                  # the agent's requests plus the speech (result.tool_reported_cost)

Return the spend marker on the error branch too. The OpenAI row above raises on a failure, but a Gemini TTS request that comes back blocked or empty (error_code "moderation_blocked", "empty_response" or "max_output_tokens") has still billed its prompt tokens, and clip["cost"] carries them. An error result without the marker leaves that spend out of the agent's cost.

For playback that starts while the audio is still being made, stream inside the tool; the model still gets one short result when the clip has finished:

from contextlib import aclosing

from doc_processing_toolkit.multimedia import asynthesize_speech_stream


@tool
async def speak_streamed(text: str) -> dict:
    """Say the text aloud to the user."""
    cost, estimated, error = 0.0, False, None
    async with aclosing(asynthesize_speech_stream("gpt-4o-mini-tts", text, output_format="pcm")) as frames:
        async for frame in frames:
            if frame["type"] == "audio":
                await PLAYER.feed(frame["data"])              # 24 kHz, 16-bit, mono PCM
            elif frame["type"] == "error":
                error = frame["error"]                        # the done frame still follows
            elif frame["type"] == "done":
                cost, estimated = frame["cost"], frame["cost_estimated"]
    if error:
        return {"spoken": False, "error": error, **tool_spend_marker(cost, estimated)}
    return {"spoken": True, **tool_spend_marker(cost, estimated)}

Images work the same way: wrap multimedia.generate_image (or edit_image), show result["images"] in your UI and return a short description with the spend marker. For multi-turn editing, ImageSession is the Python API; it has no MCP tool.

Pattern 2: the multimedia MCP server

The server's tools write each clip and image to a file and return its path. Attach the server in-process, or run it over stdio, and give the agent only the tools it needs with allowed_tools. An app that speaks through its own tool keeps the server's synthesize_speech out:

from doc_processing_toolkit import Agent
from doc_processing_toolkit.mcp_servers.multimedia_server import multimedia_server

agent = Agent(
    model="claude-sonnet-5",
    tools=[
        speak,  # pattern 1's tool
        {"type": "fastmcp", "server": multimedia_server, "allowed_tools": ["generate_image", "edit_image"]},
    ],
    tool_result_preview_chars=0,   # the events carry the whole result, so the path is never cut off
)

async for event in agent.stream("Draw a small red lighthouse for the header."):
    if event.event_type == "tool_call_result" and event.metadata["tool_name"] == "generate_image":
        for path in event.metadata["result_data"]["paths"]:
            show_image(path)                           # your UI

Over stdio the same filter goes in the command config: {"type": "mcp_command", "command": "python", "args": ["-m", "doc_processing_toolkit.mcp_servers.multimedia_server"], "name": "multimedia", "allowed_tools": ["generate_image", "edit_image"]}.

tool_result_preview_chars=0 matters here: by default a tool_call_result event carries only the first 500 characters of the result, and result_data gives you the result as data rather than Python's str() of it. The model is sent the same short result either way. See Tool results in events.

The files go to the call's output_dir, else DPT_MULTIMEDIA_OUTPUT_DIR, else <system temp>/doc_processing_toolkit/multimedia (on Linux and macOS a per-user doc_processing_toolkit-<uid> folder only that user can read). The toolkit never overwrites or deletes them, so clean the folder up on your own schedule. transcribe_audio also saves each transcript there (transcript_path) while keeping the text in its result.

Formats, and how to play or show them

Output Format Plays or shows
Google TTS (gemini-3.8-*-tts, the preview rows) WAV, 24 kHz mono 16-bit directly, in a browser <audio> element or any audio library
OpenAI TTS (gpt-4o-mini-tts) MP3 by default; opus, aac, flac, wav, pcm on request MP3 and WAV directly
output_format="pcm", and every Google stream raw 16-bit samples, no header give the player the format: 24 kHz, 16-bit, mono
An OpenAI stream the output_format you asked for (MP3 by default); ask for pcm or wav to play as it arrives as above for pcm; MP3 chunks need a decoder that accepts partial files
Images PNG by default; output_format picks JPEG or WebP where the model supports it directly; output_compression keeps files small

Letting the model look at an image

The image tools return paths, so the model sees an image only when it asks for one through a tool that returns the file as an MCP image. doc-processing-std-tools has one, read_image(path) (its documents server). Attach just that tool:

from doc_processing_std_tools.document_processing import document_processing_server

tools = [
    {"type": "fastmcp", "server": multimedia_server, "allowed_tools": ["generate_image"]},
    {"type": "fastmcp", "server": document_processing_server, "allowed_tools": ["read_image"]},
]
  • On a model that accepts images in tool results (supports_multimodal_function_responses in the registry), the image reaches the model as a real image, at image-token cost (on Claude roughly 1,600 tokens for a 1568-pixel image; OpenAI and Gemini count image tokens their own way).
  • On any other model the toolkit removes the image and puts a short notice in its place, so the model reads "image omitted" and never base64. The check sits in the agent, because an MCP server is never told which model called it.
  • Since std-tools 0.6.0 every image-returning tool scales the image it hands the model to at most 1568 px on the long edge and 3 MB (DPTTOOLS_IMAGE_MAX_EDGE_PX, DPTTOOLS_IMAGE_MAX_MB; the file on disk keeps its resolution — see guides/images), so a generated image reaches the model within Anthropic's 5 MB per-image limit without your app shrinking it. On an older std-tools read_image returned the file as it was, so keep generated images small there (size, output_format, output_compression).

In-process image tools before 0.5.1

A FastMCP server attached in-process (the way the std-tools README shows) had its tools' return values formatted by the toolkit, which turned an Image into the text <mcp.server.fastmcp.utilities.types.Image object at 0x...>. read_image, page renderers and screenshots worked only over stdio. Since 0.5.1 an in-process image, audio or content object is converted the way the server's own call_tool converts it. See MCP Integration.

Sandboxes and remote servers

  • Same machine or sandbox (an in-process server, or a stdio server the agent starts): your app and the server share the filesystem, so the paths work as they are. Inside a sandbox the temp folder is the sandbox's own.
  • App and server in different containers: point DPT_MULTIMEDIA_OUTPUT_DIR at a shared volume.
  • Provider-hosted sandboxes (Anthropic code execution, OpenAI code interpreter) are remote machines; they cannot see a local file unless you upload it.
  • A server on another machine writes to its own disk; your app needs a shared folder to read it.

What to validate

  • the tool result the model receives is short: no audio_base64 or images_base64 content, and the run's input_tokens do not jump by hundreds of thousands after a speech or image call;
  • the clip or image reaches your player or UI;
  • result.tool_reported_cost is the speech or image spend, and result.cost includes it;
  • with allowed_tools, the agent's tool list holds only the tools you listed.