Multimedia

doc_processing_toolkit.multimedia provides native text-to-speech, speech-to-text, and image generation/editing behind one registry-validated interface (OpenAI + Google only).

Capabilities

from doc_processing_toolkit.multimedia import (
    synthesize_speech,
    transcribe_audio,
    generate_image,
    edit_image,
)

speech = synthesize_speech("gpt-4o-mini-tts", "Hello there", voice="coral", output_format="mp3")
speech["audio_bytes"]       # the encoded audio
speech["cost"]              # audio_output pricing (text tokens in, audio tokens out)

text = transcribe_audio("gpt-transcribe", webm_bytes, duration_seconds=60)
text["text"]                # transcript
text["duration_minutes"]    # the clip length that drove per_minute billing
text["duration_source"]     # which rung of the precedence answered

image = generate_image("gpt-image-2", "A red circle", size="1024x1024", quality="low")
image["images"]             # list of image bytes (also written to output_dir when given)
image["cost_breakdown"]     # named cost lines that sum to image["cost"]

edited = edit_image("gpt-image-2", "Replace the circle with a square", image=img, mask=mask)

Files written to output_dir are named generated_image_YYYYMMDD_HHMMSS_ffffff_N.<ext> — a microsecond timestamp plus the image's 1-based index within the call — and the result's paths list is the authoritative record of where they went. The name carried seconds only before 0.4.0, so two calls within the same second overwrote each other's files while both reported a path; code that parses the name must accept the extra _ffffff group. The clock can still repeat a stamp (a Windows tick is 1 to 16 ms), so since 0.5.1 each file is also created exclusively, and a name that is already taken gains _2, _3 and so on before the extension. Read paths rather than predicting a name.

Optional knobs beyond the mandatory model + input:

• TTS — voice (a prebuilt name or a custom-voice object), output_format, speed, instructions and multi_speaker_config (see Text to speech); the audio can also be streamed as it is generated (Streaming speech)
• STT — diarization, timestamps, language, duration_seconds, plus custom_vocabulary, mode, languages, keywords and stream (each gated on a registry capability flag — see Speech to text)
• Images — size, quality, output_dir, background, output_format, output_compression, timeout, and aspect_ratio (Google); edit_image additionally takes n and input_fidelity, and its image= accepts a list
• Anything else — a provider_options dict passed through verbatim for provider-specific keys the facade does not name

Every new image knob is keyword-only, optional and registry-validated. edit_image used to have no quality parameter at all.

size is a provider-shaped string, not a common enum: OpenAI takes pixel dimensions ("1024x1024"), Gemini takes a resolution class ("1K", "2K", "4K" — gemini-3.1-flash-lite-image accepts "1K" only). Each model's accepted values are the supported_sizes list in the registry — except on the models that publish a dimension rule instead, where image_size_for solves it for you.

Registry-driven validation

Every option is validated against the model's capability metadata in model_information.json (descriptions.<model>) before any API call. An unsupported option raises MultimediaValidationError naming exactly what was rejected: an unknown voice, a speed value the model does not accept, diarization on a non-diarizing model, timestamps where word timings are unsupported, an invalid size, or a pixel mask passed to Google editing (Gemini image editing is instruction-driven and takes no mask). Voice names are opaque provider strings — the registry lists each model's set; there is no cross-provider enum.

The gates only fire when the metadata is there

Validation is data-driven, so a missing capability key means the gate silently does not run. The three Gemini image rows carried no modality, supported_sizes, or supports_mask_editing until 0.3.0 — an invalid size reached the API instead of being rejected locally. They now carry all three, so size="99x99" fails before any request is sent.

Choosing an image size

Sizes come from a rule, not a list. Older image models publish a short enum of preset sizes. The GPT Image 2 and 2.5 models publish a rule instead: any WIDTHxHEIGHT whose width and height are both multiples of 16, whose aspect ratio stays between 1:3 and 3:1, whose longest edge is at most 3840 px and whose pixel count is between 655,360 and 8,294,400.

multimedia.image_size_for(model, width, height) solves it and returns a value you can pass straight to size. It is a pure function — it reads the registry and makes no API call — and it answers for preset models too, returning the closest listed preset, so one helper covers every model you might route to:

from doc_processing_toolkit.multimedia import generate_image, image_size_for

size = image_size_for("gpt-image-2.5-flare", aspect_ratio="3:4", target="1K")  # '864x1152'
result = generate_image("gpt-image-2.5-flare", "a product photo", size=size, quality="low")

target is "1K", "2K" or "4K" — a total-pixel budget that overrides the one implied by width/height, and "2K" is the default when only an aspect_ratio is given. Anything above 2560x1440 is documented by the vendor as experimental: the solver returns it with a warning, never an error. A size outside the rule raises MultimediaValidationError naming the constraint it broke, so you find out which of the five rules you missed.

Rows publish the rule as descriptions.<model>.size_constraints; a row without it keeps preset behaviour. Before this release the toolkit gated size through the four-value preset enum on every model, so a resolution the API documents as valid was rejected locally. The same helper is available as an image_size_for MCP tool — see MCP.

The accepted literals are exported as constants rather than spelled out at each call site: multimedia.IMAGE_SIZE_TARGETS (the three pixel budgets), DEFAULT_IMAGE_SIZE_TARGET, IMAGE_BACKGROUNDS ("transparent", "opaque", "auto") and IMAGE_INPUT_FIDELITIES ("low", "high"). Which of them a given model accepts is still a registry question — supported_aspect_ratios (with aspect_ratio_size_constraints for a ratio a row offers at some sizes only — the Nano Banana 2.1 panoramas at 2K and 4K — refused with any other size, or none, before a request), supported_output_formats, supports_input_fidelity and supports_transparent_background gate the matching parameters, and the code names no model, so enabling a capability is a JSON-only change.

A named parameter beats provider_options

generate_image and edit_image merge both into one request. Where both set the same key the named parameter wins, because it is the one validated against the registry, and the discarded passthrough value is logged at WARNING naming both sides. provider_options remains the escape hatch for every key the facade does not name, and for a named parameter you left unset — so passing background through provider_options still works as long as you do not also pass background=.

This reverses the previous precedence on the image functions, which was documented as intended: an unvalidated passthrough silently beat a validated argument, and on Google an image_config in provider_options silently discarded an accepted size. synthesize_speech and transcribe_audio keep passthrough-wins, and the asymmetry is intended: on the image calls the named parameter wins, on the speech calls provider_options does, so existing TTS/STT callers that override a named option through the passthrough keep working.

An option the target provider has no field for is now rejected locally rather than dropped: quality, background, output_format, output_compression, input_fidelity and n > 1 are OpenAI-only, and aspect_ratio is Google-only (size by pixels with image_size_for on OpenAI). n > 1 on Google was previously accepted and ignored. A silently ignored background="transparent" is a bill for an image you cannot use.

Editing takes several reference images

edit_image(model, prompt, image=[a, b, c]) accepts a list as well as a single image — OpenAI takes up to 16 on one edit, Google takes them as parts ahead of the prompt and documents up to 14 (up to 4 characters, up to 10 objects). A mask stays OpenAI-only and, with several images, applies to the first. size is honoured on both providers now: it used to be validated everywhere and forwarded on OpenAI only, so an accepted "2K" still produced a 1K image on Gemini.

Video and PDF references (0.5.4). On a Google row whose registry input_types list "video" or "document" — gemini-nano-banana-2.1 today — a reference may be a video clip or a PDF as well as an image: "paint a still in the mood of this clip", "make a poster from page one". Every reference is sniffed for what it is (PNG, JPEG, WebP, GIF, a HEIC / HEIF / AVIF still by its ftyp brand, PDF, MP4, WebM, or by its filename for a container the sniffer does not know) and declared to the API as that type; a non-image reference on any other row — the other Gemini image rows, every OpenAI row — is refused before any request, with the registry's own message. The same references travel through the MCP tool's image_path / image_base64 / image_paths_json / images_base64_json arguments. Video and PDF tokens are input tokens at the row's input rate (a PDF page is about 258 tokens; video is billed by duration). A reference up to 20 MB goes inline in the request; a larger one goes once through Google's Files API and reaches the model as a file_data part, and edit_image deletes that upload after the call; an upload Google fails to process, or that is still processing at the deadline, is deleted before the error is raised, on either path (an ImageSession keeps its live uploads — see Image Sessions). The threshold, GOOGLE_INLINE_REFERENCE_MAX_BYTES in multimedia, is a safety margin rather than Google's cap: the file-input guide allows 100 MB inline (50 MB for a PDF), a 60 MB clip went through inline in the 9 Oct 2026 check, and the 20 MB figure older pages quote is the one the margin follows. Either route feeds the model the whole file; the Files API adds an upload and a short server-side processing wait, and has caps of its own (2 GB a file, 20 GB a project, 48-hour retention).

Asking about a media file, and asking about it locally

Two small facades sit beside the four above. describe_media packages the tool-less-Agent pattern every caller re-invents; inspect_media answers the questions that do not need a model at all.

from doc_processing_toolkit.multimedia import describe_media, inspect_media

facts = inspect_media(scan_bytes, include_models=True)
facts["kind"], facts["mime_type"], facts["size_bytes"]   # 'image', 'image/png', 184320
facts["width"], facts["height"]                          # from the PNG header
facts["accepted_by"]                                     # every registry row that takes an image

answer = describe_media("gemini-3.5-flash", clip, "What is being demonstrated?", fps=1.0)
answer["text"], answer["cost"], answer["media"]          # 'media' is the inspect_media record

inspect_media makes no network call. It sniffs the container from the payload's own bytes and reads pixel dimensions out of PNG / GIF / WebP headers and a short JPEG SOF scan, or a WAV duration out of a RIFF header, returning None for anything the header does not give away. With include_models=True it also lists every registry row whose input_types accept that kind, which is the cheap way to answer "can I send this to that model?" before you pay to find out.

describe_media validates the modality before the upload. The media's kind is checked against the model's input_types first, so a PDF sent to an image row fails locally instead of being uploaded and billed. The Google-only video knobs — start_offset, end_offset, fps, media_resolution — raise on another provider rather than being dropped silently. The result carries the same error / error_code / retryable classification as the other facades, plus cost, cost_estimated, input_tokens, output_tokens, the inspect_media record under media, and duration_s — which here is the wall-clock time the call took, not the length of the clip. Both functions share one validator, so they cannot disagree about what a model accepts.

Multi-turn image work

generate_image and edit_image are one-shot. When a conversation with an image model has to continue — generate, refine, refine again, possibly in a different process — use ImageSession, which keeps the provider's own replay state and serialises to a restorable envelope. It takes the same references and the same thinking_level as the one-shot functions.

Thinking on the Gemini image rows

The Gemini image rows think before they draw. gemini-nano-banana-2.1 does so by default (its medium level; the 9 Oct 2026 check measured 676 thinking tokens on a 26-token prompt), the 3.x rows only when asked. thinking_level= on generate_image, edit_image and ImageSession names a level from the row's supported_thinking_levels — "minimal", "medium" or "high" on 2.1 — and reaches the API as ThinkingConfig(thinking_level=...); a level the row does not list, or the parameter on an OpenAI row, is refused locally. Thinking tokens bill at the row's output_text rate and are included in output_tokens (see Image costs are reported by component).

Text to speech

synthesize_speech(model, text, ...) returns the whole clip once it exists. Five rows serve it:

Row	Provider	Notes
gpt-4o-mini-tts	OpenAI	instructions, speed, custom voices; exact usage via SSE
gemini-3.8-flash-tts	Google	GA; 130+ languages; the 3.8 request shape below
gemini-3.8-flash-lite-tts	Google	GA; 101 languages; the 3.8 request shape below
gemini-3.1-flash-tts-preview	Google	Preview; Google names the 3.8 rows as its replacement
gemini-2.5-flash-preview-tts	Google	Preview; Google names the 3.8 rows as its replacement

Each row's descriptions.<model> entry gates the options before any request: supported_voices, supported_output_formats, supports_speed, supports_instructions, supports_multi_speaker and supports_streaming (streamed speech), plus three flags added in 0.5.1: supports_speech_metadata (the Gemini 3.8 request shape below), supports_custom_voices (a {"id": ...} voice on a Google row) and supports_sse_usage (exact OpenAI usage). The agent factory's get_model_info reports all of them, so a host can tell before calling whether synthesize_speech_stream will take a row (see Discovery). The two preview rows carry release_stage: "preview" and a deprecation record with no shutdown date, which discovery reports too.

The Gemini 3.8 rows read the text verbatim

On gemini-3.8-flash-tts and gemini-3.8-flash-lite-tts the text is a transcript: everything in it is spoken. The toolkit therefore sends the other options beside the text, never inside it. The rows carry supports_speech_metadata: true, and the behaviour is keyed on that flag, not on the model names:

• instructions become the style. They ride speech_metadata.style on the text part and are not read aloud. On the preview rows they are still prepended to the text, which is how those rows are steered. On a 3.8 row that prompt would be spoken.
• Inline tags mark moments inside the text: <laugh>, <sigh>, <cough>, <breath>, <short pause>.
• Multi-speaker keeps its calling contract: a Name: line transcript plus multi_speaker_config. A 3.8 row rejects a multi-speaker request unless every text part names its speaker, so the toolkit splits the transcript into one part per turn, each with speech_metadata.speaker (and the style, when there is one). A line opens a turn when it starts with a configured name and a colon: any case (jane: is Jane), with or without a space after the colon, and longer names are tried first, so a speaker called Dr: Who is not read as Dr. A line without one continues the previous turn. Three mistakes fail locally with a MultimediaValidationError instead of the provider's 400: text before the first speaker line, a line that starts with some other short label (at most three words, a colon, then a space; most likely a misspelled speaker), and a transcript with no speaker lines at all. A URL or a sentence containing a colon is a continuation. The same split applies when the speakers are configured only through provider_options["speech_config"], and then no default single voice is sent beside them (Gemini rejects voice_config and multi_speaker_voice_config together).

dialogue = synthesize_speech(
    "gemini-3.8-flash-lite-tts",
    "Joe: How's it going today, Jane?\nJane: Not too bad. <laugh> How about you?",
    multi_speaker_config=[{"speaker": "Joe", "voice": "Puck"}, {"speaker": "Jane", "voice": "Kore"}],
    instructions="Relaxed, like two old friends.",
)

• Custom voices. Besides the 30 prebuilt voices, a 3.8 row accepts voice={"id": ...}: an Extended Voice Library id (Google's Voices API, client.voices.list(language_code=["en-US"]), lists them beside the prebuilt voices, for example en-us-advisor-1), a designed voice (voice_...) or a replicated voice (voice_... / voicekey_...). It is sent as the VoiceConfig.voice field that google-genai 2.25.0 added. It also works inside multi_speaker_config entries. The rows carry supports_custom_voices: true; a row without that flag refuses the object locally.
• The unary reply is a finished WAV (24 kHz, mono, 16-bit), and it is returned exactly as sent. The preview rows return raw PCM, which the toolkit wraps in a WAV at the sample rate the reply names. Before 0.5.1 every Google reply was wrapped, so a 3.8 WAV came back inside a second WAV header.

Output formats

output_format is validated against the row's supported_output_formats. On OpenAI it selects the encoding: mp3 (the default), opus, aac, flac, wav or pcm. The Google rows list wav and pcm:

• wav, the default, returns a WAV container.
• pcm returns the raw 16-bit samples, with mime_type audio/l16;rate=24000. Before 0.5.1, "pcm" returned a WAV container too. DPT_GOOGLE_TTS_RAW_PCM=off restores that (see Configuration).

Google's other formats (audio/mulaw, audio/alaw, 16 and 8 kHz) are documented for its Interactions API only; generate_content, which synthesize_speech uses, rejects them.

Exact OpenAI TTS cost

A plain OpenAI speech response is audio with no usage object, so the cost used to be estimated from the text length and the clip's duration. On a row with supports_sse_usage: true (gpt-4o-mini-tts), synthesize_speech now asks for the SSE form of the same response (stream_format="sse"). It assembles the audio from the speech.audio.delta events and bills the input_tokens and output_tokens of the final speech.audio.done event, so cost_estimated is False.

• If the audio arrived but the usage event did not, the audio is kept and its cost estimated from the clip, with a WARNING. Repeating the request would bill a second clip and discard this one.
• If the stream carries no usable audio (none at all, a chunk that does not decode, an error event), the request is repeated once as raw audio, with a WARNING. Both requests are billed: the result's tokens and cost are the sum of the stream's reported usage (or its estimate) and the repeat's estimate, and cost_estimated is True.
• If the server answers with plain audio anyway, that audio is used and the cost is estimated. The response's Content-Type decides which it is; without one, the first line does.
• The event stream is read by the SSE rules: comments, id and retry fields, multi-line data fields and CRLF line endings are all handled.
• A caller's own provider_options={"stream_format": "audio"} keeps the raw request for that call.
• DPT_OPENAI_TTS_SSE_USAGE=off restores the raw request and the estimate everywhere (see Configuration).

Streaming speech

synthesize_speech returns the clip once it exists. synthesize_speech_stream (a generator) and asynthesize_speech_stream (an async generator) hand back the audio while the rest is still being generated, so playback can start early: a voice UI, a phone line, an agent speaking its answer. They take the same arguments as synthesize_speech except output_path, and send the same request, so the Gemini 3.8 style, dialogue and custom-voice handling all apply.

from doc_processing_toolkit.multimedia import synthesize_speech_stream

for frame in synthesize_speech_stream("gemini-3.8-flash-lite-tts", "Welcome back.", voice="Kore"):
    if frame["type"] == "audio":
        player.write(frame["data"])           # raw 16-bit PCM, audio/l16;rate=24000
    elif frame["type"] == "done":
        print(frame["cost"], frame["total_bytes"], frame["chunks"])

• Frames. {"type": "audio", "data", "mime_type", "index"} per chunk; an optional {"type": "error", "error", "error_code", "retryable"} when the stream fails part-way; then exactly one {"type": "done", ...} with the synthesize_speech result keys except audio_bytes and output_path, plus total_bytes and chunks. The names are exported as multimedia.SPEECH_FRAME_*.
• Rows. The row must carry supports_streaming: true: gpt-4o-mini-tts, both Gemini 3.8 rows and gemini-3.1-flash-tts-preview. gemini-2.5-flash-preview-tts is refused locally. Validation runs on the first iteration, before any request.
• Formats. A Google stream is raw 16-bit PCM at 24 kHz: omit output_format or pass "pcm". "wav" is refused, because a WAV header needs the clip's total length. OpenAI streams any of its formats; pcm (24 kHz 16-bit mono) plays most easily chunk by chunk, and a streamed wav carries a placeholder length in its header.
• Cost is in the done frame, because both providers report usage only at the end: Google in the last chunk's usage_metadata (its cached share re-priced), OpenAI in the SSE speech.audio.done event on a supports_sse_usage row. A stream cannot be repeated, so a stream without a usage event is estimated (cost_estimated), and a chunk that does not decode is an error frame rather than the unary function's raw-audio retry. A request that fails before anything arrives (a 429, a refused connection) reports 0.0, as synthesize_speech reports no cost when it raises; one that drops part-way is billed from what arrived and flagged cost_estimated. An argument the SDK rejects raises instead of becoming an error frame.
• Google bills a streamed clip higher. Gemini counts each streamed chunk's audio tokens rounded up (two tokens for every 40 ms chunk on the preview row), so the same text measured on 2026-09-27 reported about 25% more output tokens per second of audio streamed than unary on both Gemini 3.8 rows, and about 55% more on gemini-3.1-flash-tts-preview. The toolkit bills what the provider reports. Stream when first-audio latency matters; use synthesize_speech for files. OpenAI reports the same usage either way.
• Stopping early closes the HTTP stream. No done frame arrives and no cost is reported (unlike Agent.stream(), which can price an abandoned run); one INFO line records the bytes received. From async code, wrap the generator in contextlib.aclosing so it closes at once.
• Not over MCP. A tool call returns one response, so the multimedia server's synthesize_speech stays unary.

Measured once on 2026-09-27, one 24-word sentence (about 9 s of audio):

Row	First audio, streamed	Whole clip, streamed	Unary
gpt-4o-mini-tts (pcm, async)	0.7 s	2.0 s	2.1 s
gemini-3.8-flash-lite-tts	1.2 s	2.8 s	3.9 s
gemini-3.8-flash-tts	3.7 s	6.8 s	4.6 s
gemini-3.1-flash-tts-preview	1.3 s	4.5 s	8.4 s

OpenAI latency on the same box varied from 1.9 s to 9.5 s between identical unary requests, and two streamed runs (one pcm, one mp3) stalled for over a minute on both the SSE and the raw path, so treat these as one sample.

How it differs from the toolkit's other streaming surfaces:

Surface	What streams	Sync / async	Cost	Stopping early
Agent.stream(), AgentSession.stream()	An LLM run, tool rounds included, as typed events	Async	On the final result	Cancels the request and reports the abandoned spend
atranscribe_audio_stream()	The transcript of a finished recording	Async	In done	Closes the stream; no done
Realtime and Live sessions	A two-way conversation over minutes	Async	Accrues over the session	close() ends it
synthesize_speech_stream(), asynthesize_speech_stream()	One text's audio, once	Both	In done	Closes the stream; no done and no cost

None of them plugs into another: an agent reaches speech only through the multimedia MCP server, whose tools are unary. To have an agent speak as it answers, consume Agent.stream() and pass each finished sentence to asynthesize_speech_stream(); the multimedia recipe shows the pattern.

Instructions are refused where a row declares none

supports_instructions: false on a TTS row now refuses instructions= locally. Before 0.5.1 nothing read the flag. Every current TTS row declares true, so nothing changes today; the flag now means what it says for future rows.

Speech to text

transcribe_audio covers three surfaces, chosen from the registry rather than from the model name: the OpenAI per-minute models (gpt-transcribe, gpt-4o-transcribe-diarize, whisper-1), Google's dedicated transcription model, and audio understanding on an ordinary audio-capable Gemini text model. Every option below is gated on a capability flag in descriptions.<model>, so passing one to a model that does not have it raises MultimediaValidationError naming the parameter that model does take, rather than dropping it silently.

The multimedia MCP tool of the same name also saves each transcript (text, segments and words) as a JSON file and returns its path as transcript_path (0.5.1), so an app keeps the whole transcript however much of the result it shows its model. The folder is the call's output_dir, else DPT_MULTIMEDIA_OUTPUT_DIR, else the system temp folder; see Files, not base64. The Python function writes no file.

gpt-4o-transcribe-diarize needs a chunking strategy past 30 seconds

OpenAI requires chunking_strategy on this model for any clip longer than 30 seconds and answers such a request with a 400 when it is missing. The toolkit never set it, so every longer diarization failed. The row now carries default_chunking_strategy: "auto", and the request builder sends it on the blocking, streamed and async-streamed paths. With "auto" the server normalises loudness and chooses chunk boundaries by voice activity. To override it, pass provider_options={"chunking_strategy": {"type": "server_vad", ...}}; your value always wins. The row also publishes supports_prompt: false, because OpenAI says prompt "is not supported when using gpt-4o-transcribe-diarize", so a prompt is refused before the upload.

A session model is not reachable on the file endpoint

per_minute says how a row is priced, not which endpoint reaches it, and this release registered duration-billed rows that only exist as a socket. transcribe_audio used to route any OpenAI per-minute row to /v1/audio/transcriptions, which for those rows is a request the API can only reject. Such a row is now refused locally, before any client is built, with a message naming the session kinds it does serve and open_realtime_session. The decision is registry data — modality and api_surfaces, never a model name — so gpt-transcribe (a stt row on two surfaces) and the other file rows are unaffected, and today the refused rows are gpt-live-transcribe, gpt-realtime-whisper, gpt-realtime-translate and gpt-live-1. For live capture, open the session those rows do serve: live transcription, live translation or GPT-Live.

Transcribing a recording with Gemini

gemini-3.5-transcribe is Google's dedicated speech-to-text model for completed recordings, reached through the same transcribe_audio call as the per-minute models. It detects 85+ locales automatically, including mid-sentence code-switching, and takes three options the per-minute models do not: custom_vocabulary for up to 1,000 domain terms, mode — "VERBATIM" (the default: word for word, fillers kept) or "SMART" (disfluencies removed; paragraphs, lists, currencies and dates formatted) — and languages, a list of BCP-47 hints where an empty list means auto-detect. It also supports diarization and timestamps.

result = transcribe_audio(
    "gemini-3.5-transcribe", meeting_bytes, diarization=True, timestamps=True
)
result["text"]                    # the full transcript
result["segments"][0]["speaker"]  # 'spk_1'
result["words"][0]["start"]       # 0.1 (seconds; "0.100s" is kept as start_offset)

The transcript is read from each candidate part's audio_transcription annotation, not from response.text — on the live API that attribute comes back empty for this model — and the same annotations carry the speaker labels and word offsets that fill segments and words.

Several combinations are documented 400s, so the toolkit rejects them locally before the recording is uploaded: custom_vocabulary cannot be combined with diarization or with word timestamps, and mode="SMART" cannot be combined with either. One request takes at most an hour of audio, or thirty minutes once diarization or word timestamps are on. That ceiling is checked only when the clip's length is actually known — from the provider, from duration_seconds, or from a WAV header — because the byte heuristic under-counts compressed audio about tenfold and would otherwise reject perfectly good recordings. Over-long recordings are rejected with a named error; the toolkit never splits a recording and stitches the pieces back together, because timestamps would need re-basing and speaker labels do not survive a split. Diarization is documented for up to 8 speakers and is experimental from 3 upward, but the SDK exposes no speaker-count field, so that limit is registry data (max_diarization_speakers) and a log line rather than a validation.

This model bills tokens, and its output tokens are estimated

gemini-3.5-transcribe is priced per token, not per minute: $2.00 per million audio-input tokens and $12.00 per million text-output tokens. The live API reports the audio-input count but no output token count at all, so the text output is estimated from the transcript itself and such a result comes back with cost_estimated: True. A response that does report a count is billed on that count and is not flagged. duration_minutes is reported when it is known but never drives the cost, and the blended per-minute figure on Google's pricing page is that page's own estimate rather than a tariff.

Audio containers and conversion

Speech-to-text providers accept a fixed list of audio containers, and a browser recording is usually WebM/Opus or MP4. The registry's audio_requirements section records what each provider accepts and which MIME type each container is sent as; gemini-3.5-transcribe accepts a wider set than Gemini audio understanding and carries its own accepted_audio_containers list. When the container sniffed off your payload is not accepted, the toolkit transcodes it and the result reports converted_from.

The converter is optional on purpose — a bundled ffmpeg is a ~25 MB wheel and most applications never call this path. The toolkit discovers one instead: DPT_FFMPEG_PATH, then ffmpeg on PATH, then the binary imageio-ffmpeg ships, then PyAV. With none of them installed, an unaccepted container raises a MultimediaValidationError naming the model, the container, the accepted list and the doc-processing-toolkit[audio] extra — rather than reaching the provider as a 400. DPT_AUDIO_CONVERT_TIMEOUT (60 seconds) bounds one transcode and DPT_AUDIO_CONVERT_MAX_MB (200) bounds the input; see Configuration. The same module exposes pcm_to_wav(), which needs no converter at all and is the cheap way to wrap a realtime capture buffer.

Before this release the MIME type sent to Gemini was audio/wav whatever the payload actually was, because the upload filename was invented before the container could be sniffed. The type now comes from the sniffed container via the registry's container_mime_types; a caller-supplied media_type still wins, and a payload that cannot be identified at all falls back to audio/wav with a warning rather than silently.

Streaming a transcript

stream=True on transcribe_audio reads the provider's streamed transcript and returns the same result dict; atranscribe_audio_stream gives you the deltas as they arrive:

from doc_processing_toolkit.multimedia import atranscribe_audio_stream

async for frame in atranscribe_audio_stream("gpt-transcribe", clip_bytes):
    if frame["type"] == "delta":
        print(frame["text"], end="")
    elif frame["type"] == "done":
        print(frame["cost"], frame["duration_source"])

Frames are delta, segment, an optional error, and exactly one done carrying the same keys transcribe_audio returns. The frame names are exported as multimedia.TRANSCRIPTION_FRAME_*.

Streaming applies to the model's output, not its input: the recording must already exist. For live capture, open a transcription session instead — audio in, transcripts out, no model turn — on gpt-transcribe, gpt-live-transcribe, gpt-realtime-whisper or gemini-3.5-transcribe-live. Both forms here are gated on the registry's supports_streaming_transcription — gpt-transcribe and gpt-4o-transcribe-diarize carry it, whisper-1 and gemini-3.5-transcribe do not — so whisper-1, where the API accepts stream and ignores it, is refused locally instead of returning nothing for a billed request. Passing stream through provider_options is an error: it used to reach the SDK, which handed back a stream object that read as an empty transcript at full price. A stream that yields no text is reported as an empty_response error with its real cost, never as a successful transcription. A final done event whose text is empty falls back to the concatenated deltas, with a warning — it used to discard them and report that billed empty_response instead.

Cost reporting

Multimedia models use three pricing schemas alongside the text schemas (see docs/concepts/cost_tracking.md):

• audio_output — TTS: text input tokens plus audio output tokens. Gemini TTS reports real token usage; since 0.5.1 its cached_content_token_count, a subset of the prompt tokens, is re-priced at the row's cached_input rate (the Gemini 3.8 rows are the first Google TTS rows to publish one) instead of the full input rate. OpenAI TTS reports it too on a row with supports_sse_usage (gpt-4o-mini-tts), read from the SSE speech.audio.done event, so cost_estimated is False; on the raw-audio path (DPT_OPENAI_TTS_SSE_USAGE=off, stream_format="audio", or the fallback) the tokens are estimated from the text and the clip duration and cost_estimated is True. See Exact OpenAI TTS cost.
• per_minute — STT: billed on the clip length, resolved through the four-rung precedence below. Not every speech model is priced this way: gemini-3.5-transcribe is a standard row carrying an input_audio rate, so it is billed from the response's token counts and duration_minutes is reported without ever driving the cost.
• multimodal_output — images: text and image output tokens priced separately (image tokens are billed even when the model's text-output price is zero, as with gpt-image-2). A response with no output breakdown is priced from the row's own rates and flagged estimated either way: a row whose output_text is 0 (the OpenAI image rows) bills the image rate, because its text rate would report a whole generation as free; a row with a real text rate (the Gemini image rows, ordinary generateContent models that mostly answer in text) bills that text rate. An earlier fix in this release billed every such row at the image rate, which over-billed a Gemini text reply 10–20x; before that, the image rows could report a generation as free with cost_estimated False.

Transcription duration: where the number comes from

A per_minute STT model bills the measured clip length, so transcribe_audio resolves it in a fixed order and tells you which source answered through duration_source:

duration_source	Rung	Sets cost_estimated?
"provider"	The provider's own measurement. whisper-1 reports it as duration on the verbose_json body the toolkit requests for every model the registry flags supports_word_timestamps; gpt-transcribe reports its billed length as a usage object of type duration, which the toolkit reads as well — that measurement used to be discarded in favour of the byte heuristic	No
"caller"	A duration_seconds you supplied	No
"wav_header"	A WAV payload's length, computed from the header's rate/width/channels against the actual byte length past the data chunk — never the header's frame count (streamed WAVs carry a placeholder chunk size that would inflate a 5-second clip into hours) and never a fixed 44-byte offset (LIST/INFO and fact chunks sit between header and samples)	No
"byte_estimate"	A byte-length heuristic	Yes
"unavailable"	Nothing could establish a length	—

Only the heuristic is flagged, and it assumes uncompressed 16 kHz/16-bit/mono PCM — a browser's 60-second WebM/Opus recording is about 180,000 bytes, which reads as 0.09 minutes instead of 1.0, a ~10.7x under-count. Pass duration_seconds whenever you know the real length and the per-minute cost becomes exact:

result = transcribe_audio("gpt-transcribe", webm_bytes, duration_seconds=60)
result["duration_minutes"]  # 1.0
result["duration_source"]   # "caller"
result["cost_estimated"]    # False

duration_seconds is keyword-only and finite/non-negative; 0 is accepted and means "unknown", so it defers to the next rung rather than billing a confident free transcription. A duration-billed request never reports a confident $0.00 — an unmeasurable clip bills a 0.02-minute floor and is flagged. Gemini audio understanding bills tokens rather than minutes, so it reports "unavailable" instead of inventing a length, and the byte heuristic is never applied there. The literal values are exported as multimedia.DURATION_SOURCE_*.

Image costs are reported by component

generate_image and edit_image return input_tokens, output_tokens and a cost_breakdown carrying cost_input_text_usd, cost_input_image_usd, cost_text_usd, cost_image_usd and cost_cache_discount_usd, which sum to cost. It is the image-side face of cost_components_out.

Two rates it now reads that it previously did not:

• Image input tokens bill at the row's input_image rate where one is published — gpt-image-2 charges $8/M against $5/M for text, so a 1,000-token prompt of 200 text + 800 image tokens used to under-bill its input component by 32%. cached_input_image is honoured on the same subtraction convention as cached audio.

• A Gemini response's text tokens bill at output_text, not at the image rate. edit_image declared every output token IMAGE instead of reading candidates_tokens_details as generate_image already did — $60/M instead of $3/M on gemini-3.1-flash-image. A response whose token detail names only TEXT — a declined generation answered in prose — bills entirely at output_text; only a response with no modality detail at all falls back to the image rate.

• Thinking tokens bill at output_text (0.5.4). usage_metadata.thoughts_token_count was never read, so the thinking a Gemini image row does before drawing went unbilled — on every call of gemini-nano-banana-2.1, which thinks by default (676 tokens, $0.005 of a $0.041 call in the 9 Oct 2026 check), and on the 3.x rows whenever a level was requested. They are added to output_tokens and priced at the text rate; a blocked candidate that carries thinking tokens and nothing else bills them the same way. No switch: it corrects a wrong number, as the earlier edit over-bill fix did.

The OpenAI Images endpoint reports no cached-token signal at all, so gpt-image-2's cached_input and cached_input_image rates are not reachable through these two functions.

Errors are classified, not just described

Every multimedia result carries error_code and retryable alongside error. A moderation block (moderation_blocked) is never retryable — re-sending it only spends money — while empty_response, max_output_tokens, timeout, connection_error, a 429 and any 5xx are. The vocabulary is moderation_blocked, empty_response, max_output_tokens, validation_error, timeout, connection_error, http_<status>, plus whatever string the provider returned.

The MCP tools return exceptions as error results with the same two fields; a Python caller that catches a provider exception itself can classify it identically with multimedia.classify_media_error(exc), so both surfaces speak one vocabulary.

A blocked text-to-speech candidate now returns no audio rather than a 44-byte WAV header, writes nothing to output_path (which comes back None), names the finish_reason / block_reason, and still reports the cost of the prompt tokens the provider billed. Blocked image responses used to be indistinguishable from transient empty ones — both returned a bare "No image data returned by the API."

Provider notes

• Google TTS: the preview rows return raw PCM, which the toolkit wraps into a playable WAV; the Gemini 3.8 rows return a finished WAV, which is passed through. output_format="pcm" returns the samples on either (see Output formats).
• A Google TTS provider_options={"speech_config": ...}, as a dict or a types.SpeechConfig, is deep-merged into the config that voice= / multi_speaker_config= build, and the passthrough wins wherever both set the same field. Before this release the two collided as duplicate keyword arguments and the call raised TypeError. There is no named language_code, because Google says "the TTS models detect the input language automatically". Set it through speech_config if you must.
• OpenAI TTS voices: gpt-4o-mini-tts now lists marin and cedar, which OpenAI recommends "for best quality". Before this release the registry rejected both. voice= also accepts an OpenAI custom-voice object {"id": "voice_..."}, which bypasses the list. The Gemini 3.8 rows accept one too (supports_custom_voices); the Gemini preview rows refuse one by name. speed is checked against OpenAI's documented 0.25–4.0 range (multimedia.TTS_SPEED_RANGE). provider_options["stream_format"] values other than "audio" are refused: "sse" returns an event stream, which synthesize_speech used to hand back as the audio bytes. The toolkit asks for the SSE form itself on rows with supports_sse_usage, and reads it (see Exact OpenAI TTS cost).
• Google STT has two shapes: the dedicated gemini-3.5-transcribe row (see Transcribing a recording with Gemini), and audio understanding on a regular Gemini text model (e.g. gemini-3.5-flash-lite) asked in a prompt to transcribe, with input_audio modality pricing. A Google row must declare audio among its input_types to be used for either: any row was previously accepted, so audio sent to an image or text-to-speech row was uploaded and billed before the provider rejected it.
• gpt-image-2 supports pixel-mask editing (transparent mask regions mark the editable area) and transparent PNG/WebP backgrounds (a preview feature on this model, confirmed live on 2026-09-20; gpt-image-2.5-flare and gpt-image-2.5-sunburst support them without the caveat), and requires a verified OpenAI organization. Transparency is gated by the registry flag supports_transparent_background, so it is a data change, not a code change.
• Gemini image models edit by instruction only — describe the change in the prompt.
• A Google image edit declares each reference's MIME type from its magic bytes (PNG, JPEG, GIF, WebP; since 0.5.4 also PDF, MP4 and WebM), falling back to the filename and then to image/png. Earlier versions declared every input as image/png, which mislabelled a JPEG or WebP source to the API.
• The Gemini image rows return JPEG on the Developer API (observed 2026-10-09 on the lite, 3.1 Flash and 2.1 rows; ImageConfig.output_mime_type exists only on Vertex, so output_format stays an OpenAI option). Since 0.5.4 a file written by output_dir, or by the MCP tools, is named from its bytes — .jpg for a JPEG, .png for a PNG — instead of .png for everything. A client that globbed *.png for Gemini output should read paths instead. The registry's output_types say the same.
• generate_image(size=...) reaches Gemini as ImageConfig(image_size=...), and edit_image(size=...) now does too. Before 0.3.0 generate_image's value passed validation and was then dropped on the Google branch; edit_image dropped it until this release, so an accepted "2K" still produced a 1K image. Both image functions build their Google config through one helper, and that helper sets only the fields the installed google-genai declares — a field this SDK cannot carry raises, naming the version, instead of vanishing. An image_config in provider_options no longer discards an accepted size; see A named parameter beats provider_options.
• A bad image argument to edit_image used to report "audio must be bytes…", because images were loaded through the audio loader; and output_dir wrote every file as .png regardless of the requested output_format. Both are fixed.

Upload filenames are derived from the content

OpenAI's images.edit and transcription endpoints read each uploaded part's content type off the filename they are given, so raw bytes and filename-less dicts cannot be uploaded under a placeholder name. The toolkit sniffs the payload and names it accordingly — image.png, mask.jpg, audio.wav, audio.mp3 and so on — keeping any caller-supplied name that already resolves to the right media type. This closed a real failure: image bytes reached images.edit as audio.wav, image.bin or mask.bin (the last two from the multimedia MCP server, whose base64 arguments arrive nameless) and the API rejected them. Audio sent through that server was rejected the same way as audio.bin; its container is now sniffed to .wav / .mp3 / .ogg / .flac / .webm / .m4a.

The same fabrication used to happen one level earlier, on the ordinary Python API: bare bytes handed to transcribe_audio were labelled audio.wav before the sniff could run, and a name that already carries a valid audio extension is kept as-is — so every MP3 passed as raw bytes uploaded as a WAV and OpenAI rejected it with a 400 unsupported_format. Filename-less input now reaches the sniffer unnamed. Pass {"data": ..., "filename": "clip.mp3"} when you want to name it yourself.

Related pages

• Speech, Transcription and Media Questions recipe — every call on this page, run end to end with its cost
• Image Sessions — multi-turn image generation and editing that survives the process that started it
• Realtime Voice — the session-oriented surface for speech-to-speech conversation, where a connection is held open across turns instead of one request per call. It also covers the three dedicated session families: live transcription, live translation (speech in, translated speech out, no turns) and GPT-Live
• MCP — the same four capabilities exposed as MCP tools
• Cost Tracking — the audio_output, per_minute, and multimodal_output pricing schemas