Skip to content

Multimodal Content

The toolkit supports passing images, PDFs, video, and audio to models across all four providers. The MediaTransformer handles conversion of file paths, URLs, bytes, and pre-encoded content to the format required by each provider's API.

Provider support matrix

Content type Anthropic OpenAI Google Fireworks
Images ✅ ✅ ✅ ✅
PDFs ✅ ✅ ✅ ❌
Video ❌ ❌ ✅ ❌
Audio ❌ ❌ ✅ ❌

The matrix above is the provider answer. Whether a particular model accepts a particular part is a registry question — descriptions.<model>.input_types — and it is checked locally before any request is built. See Unsupported parts are now loud.

Supported file formats

Category Formats
Image PNG, JPEG, GIF, WebP, BMP
Document PDF, plain text, CSV
Video MP4, MPEG, WebM, QuickTime
Audio WAV, MP3, OGG, FLAC

Passing files to agent.run()

Files are passed via the files parameter, which accepts several input types.

File paths

Local file paths (string or Path object):

result = await agent.run(
    "Summarize this document",
    files=["./report.pdf"],
)

Multiple files of different types can be passed in a single call:

result = await agent.run(
    "Describe these images and extract text from the PDF",
    files=["./photo.jpg", "./diagram.png", "./contract.pdf"],
)

URLs

HTTP/HTTPS URLs are downloaded automatically:

result = await agent.run(
    "Describe this image.",
    files=["https://example.com/photo.jpg"],
)

Raw bytes

Pass raw bytes objects directly. The MediaTransformer auto-detects the media type from magic bytes when possible. If detection fails, provide media_type via kwargs:

with open("chart.png", "rb") as f:
    image_bytes = f.read()

result = await agent.run(
    "Analyze this chart.",
    files=[image_bytes],
)

Pre-structured dicts

Pass a dict with data (bytes or base64 string) and media_type. Use the actual MIME type of the content (e.g., image/jpeg for JPEG files, image/png for PNG):

import base64

with open("diagram.png", "rb") as f:
    encoded = base64.b64encode(f.read()).decode("utf-8")

result = await agent.run(
    "Explain this diagram.",
    files=[{"data": encoded, "media_type": "image/png"}],
)

Or pass a dict with raw bytes:

result = await agent.run(
    "Describe this image.",
    files=[{"data": image_bytes, "media_type": "image/jpeg", "filename": "photo.jpg"}],
)

Filename preservation

When you supply a filename (via dict input, path-derived from a local file, or as a filename= kwarg on raw bytes), the toolkit preserves it through to each provider's API:

  • Anthropic and OpenAI: the tool_files upload loop sends a (filename, bytes, content_type) multipart tuple to the Files API so the original filename is recorded instead of the SDK default. (Before 0.5.2 the OpenAI tool loop, the path a run with local tools takes, never completed that upload: the files did not reach the container and the run went on without them, with an error in the log.)
  • Google: the filename is used as the Files API display_name on client.files.upload, replacing the sequential tool_file_{i} placeholder.

This matters most for tool_files (uploads consumed inside Anthropic code execution / OpenAI code interpreter / Google code execution), where multiple unnamed uploads would otherwise collide on a default filename inside the container. Pass a distinct filename for each tool_files entry to keep them addressable from inside the container by their original names.

Mixing input types

All input types can be combined in a single call:

result = await agent.run(
    "Compare these files.",
    files=[
        "./local_report.pdf",
        "https://example.com/chart.png",
        image_bytes,
    ],
)

Media resolution (Google)

Google models support configurable media resolution at two levels:

Global resolution

Set a default resolution for all media using the media_resolution parameter on the Agent constructor:

agent = Agent(
    model="gemini-2.5-flash",
    media_resolution="MEDIA_RESOLUTION_HIGH",
)

result = await agent.run(
    "Describe this image.",
    files=["./photo.jpg"],
)

Per-file resolution

Override resolution per file using the media_resolution_list parameter on agent.run(). Provide one entry per file in files, in the same order:

agent = Agent(model="gemini-2.5-flash")

result = await agent.run(
    "Describe these images.",
    files=["./photo.jpg", "./diagram.png"],
    media_resolution_list=["MEDIA_RESOLUTION_HIGH", "MEDIA_RESOLUTION_LOW"],
)

Use None for files that should use the global/default resolution:

media_resolution_list=["MEDIA_RESOLUTION_HIGH", None, "MEDIA_RESOLUTION_LOW"]

Supported resolution values

Value Description
MEDIA_RESOLUTION_UNSPECIFIED Default setting (varies by model)
MEDIA_RESOLUTION_LOW Lower detail, faster processing, lower cost
MEDIA_RESOLUTION_MEDIUM Balance of detail, cost, and latency
MEDIA_RESOLUTION_HIGH Higher detail (recommended for most use cases)

Tip

Use MEDIA_RESOLUTION_LOW for simple images to reduce token usage. Use MEDIA_RESOLUTION_HIGH for images where detail matters (charts, fine text, diagrams).

Unsupported parts are now loud

A part the model cannot accept — an image sent to a text-only model, a video sent to a model without video input, a PDF sent to Fireworks (document inlining is deprecated there, and no Fireworks registry row declares a document input type) — is still dropped with a warning by default, so behaviour is unchanged.

What is new is that every drop is recorded on AgentResult.dropped_parts, one entry per part:

result = await agent.run("Summarise this.", files=["./scan.pdf"])

for part in result.dropped_parts:
    part["kind"]      # e.g. 'base64-application/pdf'
    part["index"]     # its position in the prompt, so you can name it to a user
    part["reason"]    # why the model would not take it
    part["provider"]  # and which model was asked
    part["model"]

A host can therefore detect the drop without scraping logs. Agent(on_unsupported_part="raise") refuses the run instead, with UnsupportedPartError (a ValueError subclass, exported from the package root) raised before any request is sent — the error carries the same list, so a caller learns about every offending part rather than only the first. The policy has a YAML key and mirrors on the agent factory's create_agent / create_agent_session, and the list is populated under both policies, including on an error return.

A session restored from a checkpoint gets the same check for the history it loads: a stored part the target model cannot accept is replaced by a placeholder and recorded on session.dropped_parts, or the restore raises under "raise" (History and Checkpointing).

A category the provider cannot take at all is refused earlier

on_unsupported_part governs parts a model does not list in its input_types. A category the provider has no representation for at all — video to Anthropic, say — is still refused by the media transformer with a plain ValueError before this policy runs.

Token counting for multimodal content

Each provider counts multimodal tokens differently:

  • Anthropic: Images are counted based on dimensions (tokens ≈ width × height / 750).
  • OpenAI: Images are counted based on tile breakdown at the chosen detail level.
  • Google: Uses the model's built-in token counting for all media types. Media is uploaded to the Files API first; an upload that is not yet ACTIVE when the pre-call count runs (video, typically) is excluded from that call and approximated locally, while every ready part is still API-counted.
  • Fireworks: Image tokens vary by model.

Token counts for multimodal inputs are included in the input_tokens field of AgentResult.