Skip to content

Recipe: Multimodal Content

Goal

Pass images, PDFs, video, and audio to models using the files parameter on agent.run().

Supported file types

Category Formats Anthropic OpenAI Google Fireworks
Image PNG, JPEG, GIF, WebP, BMP ✅ ✅ ✅ ✅
Document PDF ✅ ✅ ✅ ❌
Video MP4, MPEG, WebM, QuickTime ❌ ❌ ✅ ❌
Audio WAV, MP3, OGG, FLAC ❌ ❌ ✅ ❌

Passing images

from doc_processing_toolkit import Agent

agent = Agent(
    model="claude-sonnet-5",
    instructions="Describe images in detail.",
)

result = await agent.run(
    "What do you see in this image?",
    files=["./photo.jpg"],
)
print(result.output)

Multiple images:

result = await agent.run(
    "Compare these two diagrams.",
    files=["./diagram_v1.png", "./diagram_v2.png"],
)

Passing PDFs

agent = Agent(
    model="gpt-4o",
    instructions="Extract key information from documents.",
)

result = await agent.run(
    "Summarize the main findings in this report.",
    files=["./quarterly_report.pdf"],
)
print(result.output)

Mixed file types

result = await agent.run(
    "Describe the images and extract text from the PDF.",
    files=["./photo.jpg", "./diagram.png", "./contract.pdf"],
)

Video and audio (Google only)

agent = Agent(
    model="gemini-2.5-flash",
    instructions="Analyze media content.",
)

# Video
result = await agent.run(
    "Summarize what happens in this video.",
    files=["./presentation.mp4"],
)

# Audio
result = await agent.run(
    "Transcribe this audio recording.",
    files=["./meeting.mp3"],
)

URLs

Pass HTTP/HTTPS URLs directly — the toolkit downloads them automatically:

result = await agent.run(
    "Describe this image.",
    files=["https://example.com/photo.jpg"],
)

Raw bytes

Pass bytes objects when you already have file content in memory. The media type is auto-detected from magic bytes:

with open("chart.png", "rb") as f:
    image_bytes = f.read()

result = await agent.run(
    "Analyze this chart.",
    files=[image_bytes],
)

Pre-structured dicts

Pass a dict with data and media_type for pre-encoded content. Use the actual MIME type of the content (e.g., image/jpeg for JPEG files, image/png for PNG):

import base64

with open("diagram.png", "rb") as f:
    encoded = base64.b64encode(f.read()).decode("utf-8")

result = await agent.run(
    "Explain this diagram.",
    files=[{"data": encoded, "media_type": "image/png"}],
)

Mixing input types

All input formats can be combined in a single call:

result = await agent.run(
    "Compare these files.",
    files=[
        "./local_report.pdf",           # file path
        "https://example.com/chart.png",  # URL
        image_bytes,                      # raw bytes
    ],
)

Media resolution (Google)

Google models support configurable media resolution to balance detail against token usage.

# Global resolution (all media)
agent = Agent(
    model="gemini-2.5-flash",
    instructions="Analyze images.",
    media_resolution="MEDIA_RESOLUTION_HIGH",
)

result = await agent.run(
    "Describe this image.",
    files=["./photo.jpg"],
)

Or set per-file resolution:

agent = Agent(
    model="gemini-2.5-flash",
    instructions="Analyze images.",
)

result = await agent.run(
    "Describe both images.",
    files=["./photo.jpg", "./diagram.png"],
    media_resolution_list=["MEDIA_RESOLUTION_HIGH", "MEDIA_RESOLUTION_LOW"],
)

Tip

Use MEDIA_RESOLUTION_LOW for simple images to reduce token usage. Use MEDIA_RESOLUTION_HIGH for images where detail matters (charts, fine text, diagrams).

What to validate

  • result.error is None
  • result.output references the content of the provided files
  • result.input_tokens reflects the multimodal token count