Realtime Voice¶

doc_processing_toolkit.realtime holds a speech-to-speech session open across turns: you stream microphone PCM in, and audio, transcripts, and usage stream back as they are produced. It is the session-oriented counterpart to Multimedia, which does one-shot TTS, STT, and image work over ordinary request/response calls.

The public surface is provider-neutral. It is not a re-export of OpenAI's wire protocol: a session is connect → configure → send → events → close, and each adapter maps its provider down to that shape. Neither provider's event names, message shapes, nor reconnect semantics leak out. Two sibling modules add a session family each behind that same shape — doc_processing_toolkit.realtime_translation (interpreter sockets on both providers) and doc_processing_toolkit.realtime_gpt_live (OpenAI's GPT-Live family) — and both register themselves into the same routing table, so open_realtime_session(model, session_kind=...) reaches them without your importing either.

from doc_processing_toolkit.realtime import open_realtime_session

async with await open_realtime_session(
    "gpt-realtime",
    voice="marin",
    instructions="You are terse. Answer in at most three words.",
    output_format="audio/pcm",
) as session:
    await session.send_text("Say the word hello, and nothing else.")
    async for event in session.events():
        if event.type == "audio_delta":
            speaker.write(event.audio)          # raw PCM16, already base64-decoded
        elif event.type == "transcript_delta":
            print(event.text, end="")
        elif event.type == "response_complete":
            break

print(session.cost, session.usage.as_dict())    # final after close

open_realtime_session picks the adapter from the model's provider in model_information.json, so callers never name one. Its keyword arguments beyond model are optional, and any RealtimeSession.configure option (output_format, turn_detection, modalities, input_transcription, thinking_level, context_compression, max_video_fps, …) can be passed through as well. It also takes tools=, hooks=, max_tool_rounds= and tool_timeout_seconds= — see Tools in a realtime session.

Which kind of session¶

A socket is not always a conversation. open_realtime_session(model) opens a voice conversation; the same function opens each of the three dedicated families with session_kind=, and each of those has a call of its own under a clearer name.

`session_kind`	What the socket does	Own name	Detail
`"realtime"`	A voice conversation — audio and text both ways, tools, interruption	`open_realtime_session(model)`	this page
`"transcription"`	Captions only: audio in, transcripts out, no model turn	`open_transcription_session(model)`	Live transcription
`"translation"`	An interpreter: speech in, translated speech and both transcripts out	`open_translation_session(model, target_language=…)`	Live translation
`"live"`	OpenAI's GPT-Live family on its own `/v1/live/*` endpoints, billed by the second of active session time as the provider reports it	`open_live_session(model)`	GPT-Live

from doc_processing_toolkit.realtime import (
    open_realtime_session,
    open_transcription_session,
    open_translation_session,
)
from doc_processing_toolkit.realtime_gpt_live import open_live_session

voice = await open_realtime_session("gpt-realtime-2.1-mini")                      # unchanged
captions = await open_transcription_session("gpt-transcribe")                     # same call, own name
same = await open_realtime_session("gpt-transcribe", session_kind="transcription")
interpreter = await open_translation_session("gpt-realtime-translate", target_language="es")
duplex = await open_live_session("gpt-live-1")

Which kinds a model serves is registry data: get_model_info(model)["session_kinds"] reports it and list_available_models(session_kind=...) filters on it, so asking a captions model to hold a conversation fails immediately — naming what it does serve — rather than opening a socket that will never speak. resolve_realtime_session_kind(model, kind) answers the same question on its own, and every rejection (an unknown kind, a kind the model does not serve, a kind with no adapter for that provider) happens before the socket opens. session_kind=None resolves to the conversational kind wherever the model serves one, so every existing call site is unchanged.

A conversational row is not a dedicated session of another kind

session_kinds reports transcription on every conversational realtime row, because that row's own session transcribes the audio it hears. That is a derived statement about the conversational socket, not a second session family — and opening it as one is refused: open_transcription_session("gemini-3.8-live") raises, naming open_realtime_session(model, input_transcription=True) as the way to get transcripts out of a conversation. A dedicated kind is accepted on a row that also serves the conversational kind only when the row declares that kind in session_kinds.

The refusal is worth its own rule rather than a note. The dedicated adapters configure a session a conversational model rejects — a live run had gemini-3.8-live closing with a 1007 on the text-only response a captions session asks for — or price it from the wrong row, as the same call did on OpenAI. Rows that declare a dedicated kind (gpt-transcribe, gpt-live-transcribe, gpt-realtime-whisper, gemini-3.5-transcribe-live, both translation rows, gpt-live-1) are unaffected, and the derived discovery statement is unchanged.

SESSION_KIND_REALTIME, SESSION_KIND_TRANSCRIPTION, SESSION_KIND_TRANSLATION and SESSION_KIND_LIVE are exported as constants. open_live_session lives in doc_processing_toolkit.realtime_gpt_live with the rest of that family's surface; everything else here is importable from doc_processing_toolkit.realtime.

Models¶

Model	Provider	Audio in	Audio out	Voices
`gpt-realtime`	OpenAI	24 kHz PCM16	24 kHz	10 (`alloy`, `marin`, `cedar`, …)
`gpt-realtime-2`, `gpt-realtime-2.1`, `gpt-realtime-2.1-mini`, `gpt-realtime-1.5`	OpenAI	24 kHz PCM16	24 kHz	10
`gemini-3.1-flash-live-preview`	Google	16 kHz PCM16	24 kHz	30 (`Kore`, `Puck`, `Zephyr`, …)
`gemini-3.8-live`, `gemini-3.8-live-extended-thinking`	Google	16 kHz PCM16	24 kHz	no published list — see below

Four further rows open a transcription socket only — no voice, no tools, no model turn. They are listed under Live transcription. Two more open a translation socket (gemini-3.5-live-translate-preview, gpt-realtime-translate — see Live translation), and gpt-live-1 opens a live session on a protocol of its own (GPT-Live).

Two rows publish no voice list, deliberately

Google documents the 3.8 Live models as accepting "any of the voices available for our TTS models" and enumerates none, so those rows leave supported_voices unset and record why in a _supported_voices note. Copying the 3.1 list across would make the local validator reject voices the API accepts — failing a legal request offline, which is worse than not validating.

Three rows carry a retirement record

gpt-realtime retires on 2027-01-20, replaced by gpt-realtime-2.1. gemini-3.1-flash-live-preview has no announced shutdown date but Google's capabilities table calls it a legacy preview and recommends gemini-3.8-live. gemini-3.5-live-translate-preview carries one with neither a date nor a replacement, which is how a preview row says it is one. All three are recorded as a deprecation object ({date, replacement, note}) on the registry row, where a null date means "a replacement is recommended, no shutdown announced". Discovery surfaces it — see Client Integration — so you can react to a retirement without reading a changelog.

The input sample rates differ

OpenAI Realtime takes 24 kHz in both directions; Gemini Live wants 16 kHz in and emits 24 kHz out. This is the one place the neutral surface cannot paper over the providers — send_audio forwards your bytes as-is and neither provider resamples for you. Read the rate off the constants rather than hardcoding it:

from doc_processing_toolkit.realtime import (
    OPENAI_REALTIME_SAMPLE_RATE,      # 24000
    GOOGLE_LIVE_INPUT_SAMPLE_RATE,    # 16000
    GOOGLE_LIVE_OUTPUT_SAMPLE_RATE,   # 24000
)

The registry also carries input_audio_sample_rate / output_audio_sample_rate per model, which is the right source when you support both.

Event vocabulary¶

session.events() is an async iterator of RealtimeEvent. The full neutral vocabulary:

`event.type`	Meaning	Populated fields
`audio_delta`	A chunk of model speech	`audio` (raw PCM16 bytes)
`transcript_delta`	A fragment of the spoken transcript	`text`, `role`
`transcript_complete`	One committed stretch of audio has a settled transcript	`text`, `role`, `item_id`, `duration_seconds`
`transcript_failed`	The provider could not transcribe that item	`item_id`, `error`
`text_delta`	A fragment of text output	`text`, `role`
`response_complete`	The model finished answering	`usage`, `cost`
`turn_complete`	The turn boundary, on models where that is not the same thing	`usage`, `cost`
`tool_call`	The model asked for a tool	`tool_name`, `tool_call_id`, `tool_arguments`
`tool_result`	The toolkit answered it	`tool_name`, `tool_call_id`, `tool_result`, `error`
`tool_call_cancelled`	The server withdrew a tool call	`tool_call_id`
`interrupted`	The model's turn was cut short by the caller speaking	—
`session_resumed`	A transparent reconnect happened (Google only)	—
`session_restarted`	A reconnect had no resumption handle to send, so the new session has no memory of the conversation (Google only)	—
`error`	The provider reported an error	`error`

Every event also carries raw, the untranslated provider event, for callers who need something the neutral layer does not model, and item_id, the provider's identifier for the assistant item being spoken — the correlation key truncate_response() uses. role is "assistant" for model output and "user" for transcripts of the caller's own audio — the latter only appear when input_transcription=True is configured.

The module exports each type as a constant (EVENT_AUDIO_DELTA, EVENT_RESPONSE_COMPLETE, …) so a consumer can match on those instead of bare strings.

A GPT-Live session adds three of its own, defined in doc_processing_toolkit.realtime_gpt_live because only that family can produce them: delegation_created (the model handed work to your application or to the backend — the delegation id is in tool_call_id, the target in text), usage_updated (a cumulative duration snapshot in duration_seconds, never a delta) and session_closed, which is that protocol's terminal event. A translation session needs no new type at all: both transcripts arrive as transcript_delta told apart by role, and RealtimeEvent.language carries the BCP-47 code the provider attached, where it attaches one.

Configuration is validated against the registry¶

configure() stages options and checks them against the model's descriptions.<model> row before any network activity. An unsupported voice or output format raises RealtimeValidationError naming what the model does support, rather than failing mid-session:

session.configure(voice="Kore", modalities=["audio"], input_transcription=True)

Every argument is optional and only the ones supplied change, so it can be called repeatedly to adjust one knob at a time. When configuration is applied differs by provider:

A native-audio Gemini Live model will not answer in text only

gemini-3.8-live rejects a TEXT-only response modality outright — its row lists both text and audio among its output_types because it emits both, not because you may ask for one. Do not configure modalities=["text"] against it; if you want the words rather than the speech, read transcript_delta off the audio turn, or open a transcription session instead.

OpenAI applies staged options to the live session at the next send or event poll.
Gemini Live fixes its configuration at connect time. A post-connect configure() logs a warning and takes effect on the next reconnect.

configure() also takes thinking_level, proactive_audio, enable_affective_dialog, context_compression, tool_call_behavior and tool_response_scheduling. Each is checked against the model's row before the socket opens, and so is anything you set through provider_options, which reaches LiveConnectConfig verbatim — validating only the neutral path would have left the documented escape hatch wide open. session.capability(name) answers one capability with its documented default, session.capabilities returns the whole record, and realtime_capabilities(model) answers for a model you have not connected to yet. That record deliberately separates provider_capabilities (what the registry says the API offers) from toolkit_features (what this adapter actually implements), because a capability flag is not a promise that the toolkit drives it.

toolkit_features also says which session families the toolkit serves on that provider: transcription_sessions, translation_sessions and live_sessions are True exactly when an adapter is registered for that (provider, kind) pair — so live_sessions is True on OpenAI and False on Google, and a discovery client can tell "no adapter yet" from "the model cannot do it" without opening a socket. The table is checked against the routing table itself, so a routable kind cannot ship without its key.

The rest of toolkit_features follows the model's own session kind, not just its provider. The provider map is the conversational default, and the adapter for another kind overrides the keys it answers differently: gpt-live-1 reports explicit_response_cancel, response_truncation, visual_input and video_frames as False, and the two translation rows also report text_turns, tool_dispatch, async_tool_dispatch and context_window_compression as False — the controls those adapters refuse. Keyed by provider alone, the report used to claim them. Conversational rows report exactly what they did before.

`turn_detection` is translated, not forwarded¶

turn_detection is a neutral dict. None keeps the provider default (server-side voice activity detection), while {} or {"disabled": True} requests manual turn control. With manual control you close each input turn yourself, and commit_audio() does that on both providers:

await session.send_audio(pcm_chunk)
await session.commit_audio()      # only needed when turn detection is disabled

The wire messages differ. OpenAI commits the input buffer and asks for a response. On Gemini Live the client has to mark the turn itself once automatic activity detection is off, so the first send_audio() of a turn is preceded by activity_start and commit_audio() sends activity_end. The model answers the turn, and the next send_audio() opens a new activity. With automatic detection on, Gemini's commit_audio() sends audio_stream_end instead, which tells the server the microphone paused and makes it flush buffered audio. Google documents the two paths as mutually exclusive: activityStart / activityEnd only with automatic detection disabled, and audioStreamEnd only with it enabled. The adapter picks the path from your turn_detection setting. session.manual_turn_control reports which path is in force. Before this release the Gemini conversational adapter had no commit_audio() at all, and nothing sent activity_start / activity_end, so a manual-turn Gemini session could never close a turn.

gpt-live-transcribe and gpt-realtime-whisper publish supports_turn_detection: false. OpenAI says those models "require turn detection to be omitted or set to null". A turn_detection that asks for voice-activity detection is therefore refused locally with a RealtimeValidationError. {"disabled": True} is still accepted and sent as null.

Everything else is mapped per provider by translate_turn_detection, which is public and testable on its own:

Neutral key	OpenAI	Google
`disabled: True` / `{}`	`turn_detection: null`	`AutomaticActivityDetection(disabled=True)`
`type`	verbatim (`server_vad` default)	dropped → "automatic activity detection"
`threshold`	verbatim	dropped → `start_of_speech_sensitivity`
`prefix_padding_ms`, `silence_duration_ms`	verbatim	on `AutomaticActivityDetection`
`idle_timeout_ms`	verbatim	dropped → `silence_duration_ms`
`create_response`, `interrupt_response`	verbatim	dropped → `activity_handling`
`eagerness`	verbatim (`semantic_vad`)	dropped → `end_of_speech_sensitivity`
`start_of_speech_sensitivity`, `end_of_speech_sensitivity`	dropped → `threshold` / `eagerness`	on `AutomaticActivityDetection`
`activity_handling`, `turn_coverage`	dropped	on `RealtimeInputConfig`, one level up

A key the target provider cannot express is dropped with a warning naming the nearest control it does have, never approximated: threshold and start_of_speech_sensitivity are differently scaled knobs, and silently changing barge-in behaviour is worse than not applying a setting the provider never had.

Before this release the neutral dict was forwarded into AutomaticActivityDetection verbatim. That object is extra='forbid', so threshold=0.5 raised a pydantic ValidationError at connect time, and activity_handling / turn_coverage were being sent to the wrong object entirely.

Tools in a realtime session¶

A realtime model asks for a tool the same way an ordinary one does, but neither protocol runs it for you: the host executes the tool and sends the result back over the socket. Pass tools= to open_realtime_session and the toolkit does that for you, using the same tool formats, the same MCP execution path and the same hook events (tool_call_start, the before_tool_call veto, tool_call_result) as an ordinary Agent run. You see the exchange as two neutral events, tool_call and tool_result, and you can ignore both if you only care about the answer. The tool_call_result hook event carries the first 500 characters of the result and, since 0.5.1, result_truncated; a session has no tool_result_preview_chars setting, and the whole result is on the tool_result event. A session does not read the _dpt_spend marker a tool can put in its result (the multimedia MCP tools do): the marker reaches the model unread, and session.cost does not include that spend, unlike an Agent run's (see Tool-reported spend).

async with await open_realtime_session(
    "gpt-realtime-2.1-mini",
    tools=["tools/my_tools.py"],
    hooks=my_hooks,
    max_tool_rounds=8,
) as session:
    ...

Dispatch is bounded in the two ways a voice session can run away. max_tool_rounds (default 16) caps how many calls one session may run; tool_timeout_seconds (default 30) caps one call. A breached ceiling, a vetoed call, a timeout and a crashing tool all send the model an error result — a realtime model left waiting on an unanswered tool slot simply stops talking, which is a worse failure than a tool that says it failed. tools= requires supports_tool_calls on the row, and passing hooks= without tools= warns, because there is then nothing for them to observe.

On OpenAI the tool result goes out as a function_call_output conversation item as soon as the tool finishes. The follow-up response.create cannot go out yet — the response that made the call is still active, and asking for a second one is an API error — so it is sent on that response's response.done, and one request covers every call the response made. The consequence worth knowing: that calling response's completion is not surfaced as response_complete, so the event keeps meaning what it means in the text loops, "the model finished answering". A calling response cancelled by a barge-in gets no automatic follow-up at all; the result is carried by the next user turn.

On Gemini Live the behaviour is per model. gemini-3.1-flash-live-preview is blocking: it will not speak until the answer arrives, so the whole batch is answered in one send_tool_response. The 3.8 rows call tools asynchronously — the model keeps speaking and keeps accepting your speech while the tool runs. The toolkit follows the model's registry row rather than a flag you set: gemini-3.8-live defaults to NON_BLOCKING, and gemini-3.8-live-extended-thinking supports nothing else. Results are returned as they land, each carrying a scheduling hint as a top-level FunctionResponse field (WHEN_IDLE by default; INTERRUPT to cut off what is being said), and outstanding calls are cancelled when the session closes, when it reconnects — a resumed connection has no memory of the call ids — and when the server sends a tool-call cancellation, which arrives as a tool_call_cancelled event. A result that had already finished and was waiting to be sent when the reconnect happened is discarded too, rather than answered on the resumed session under an id it never issued. close() also finalizes the receive loop and its background tasks, within a two-second bound; under NON_BLOCKING they used to outlive the session.

Showing the model what you are looking at¶

A realtime session can take pictures as well as sound. session.send_image(png_bytes) attaches a still — a photographed page, a chart, a screenshot — and the model answers about it on your next turn. session.send_video_frame(jpeg_bytes) is for a continuous capture: a camera, a screen share, a document under a webcam.

The two are deliberately different. A still is a discrete act, like typing a question, so it goes on the wire immediately. A frame is one of many, and the model re-reads every one of them as billable image tokens — so frames arriving faster than max_video_fps (one per second by default) are coalesced: the newest frame is kept, the one it superseded is counted in session.frames_dropped, and nothing is queued.

That is the point rather than a limitation to work around. A 30 fps loop would otherwise multiply your bill roughly thirtyfold for a picture that barely changed, and a model asked about a screen share should answer about the screen now, not about a backlog of stale frames. Nothing ever blocks your producer, and audio is never gated or dropped. Set max_video_fps=0 if you really do want every frame.

async for frame in camera:                      # 30 fps in
    await session.send_video_frame(frame)       # ~1 fps reaches the model
await session.send_text("What am I holding?")
print(session.frames_sent, session.frames_dropped)

send_video_frame returns whether the frame went on the wire now; flush_pending_frame() releases a held frame, and both adapters call it from send_audio, so a live session's continuous audio is a free heartbeat for a capture loop that has gone still. DEFAULT_MAX_VIDEO_FPS is the default, and no registry row carries the key — no vendor publishes a frames-per-second figure.

Pass encoded image bytes (PNG, JPEG, GIF, WebP, BMP — REALTIME_IMAGE_MIME_TYPES), not raw pixels. The container is identified from the payload's own first bytes with sniff_image_media_type, and a payload the toolkit cannot identify is refused rather than labelled with a guess, so a mislabelled frame fails locally instead of being billed and rejected by the provider. Size is checked against the registry's image_requirements, and a model whose row declares neither image nor video in input_types refuses before anything is sent.

	OpenAI Realtime	Gemini Live
Still	`conversation.item.create` with an `input_image` part	`send_realtime_input(video=…)`
Frame	The same wire — a frame is an image on this protocol	`send_realtime_input(video=…)`
`detail` (`REALTIME_IMAGE_DETAILS`)	Sent verbatim	No equivalent — dropped with a warning naming `media_resolution`

On Gemini Live a still also goes out on the video channel

Google documents exactly one way to show a Live model a picture — "video frames are sent as individual images" — and a live run on gemini-3.8-live confirmed it is the only one that works: the same PNG sent on the SDK's generic media= channel was accepted without any error and then never seen, while video= answered about it correctly. So both stills and frames use video=, and a still differs from a frame only in never being rate-gated.

A frame never triggers a response on its own. It is context for whatever you say next; asking the model to answer every frame would make a capture loop talk over itself.

Pixel dimensions are not checked

image_requirements also carries max_pixels and max_edge, but reading them needs an image decoder the toolkit deliberately does not depend on. Only max_size_mb is enforced, so a frame inside the byte ceiling but over the edge limit still reaches the provider.

Knowing when a turn is over¶

The usual way to end a realtime turn is to break on response_complete. On gemini-3.8-live-extended-thinking that is not enough on its own: the model keeps reasoning in the background after the turn boundary, so turnComplete no longer means the session is idle. The toolkit reads the server's interaction_status for you on rows that publish it — you get a turn_complete event at the boundary and response_complete only once the model reports IDLE.

Every other model behaves exactly as before: turn_complete is emitted only on rows that use interaction_status, so the other streams are unchanged, and a session where the server never sends the field still completes on the turn boundary. break on response_complete therefore cannot hang.

Interrupting, and what the model thinks it said¶

cancel_response() stops the model mid-answer. truncate_response() does the other half: it tells the provider how much of that answer the listener actually heard, so the next turn is grounded in reality rather than in an answer that was cut off. The toolkit counts the assistant audio it handed you per item (session.emitted_audio_ms, reset on a new item_id and on interrupted) and uses that by default; if your application buffers audio before playing it, pass your own played_ms.

await session.cancel_response()
await session.truncate_response(played_ms=1_800)   # what the listener really heard

OpenAI supports both controls. Gemini Live has no truncation message at all, so truncate_response() raises there naming the provider; its interrupt is available only on the 3.8 rows (it is send_client_content(turns=None, turn_complete=True), which requires supports_client_content_mid_session). On 3.1 Live you interrupt by speaking, which is what voice-activity detection is for, and the error says so.

Session length: two different mechanisms¶

Session resumption and context-window compression are often described together, and they are not the same thing. Resumption reconnects a dropped transport and replays the handle — the toolkit does it for you and you see a single session_resumed event. It does not enlarge the context. Compression is the separate policy that decides what to keep as the conversation grows, enabled with configure(context_compression=...) on rows that carry supports_context_window_compression (True, or a dict with trigger_tokens / sliding_window_target_tokens).

session.compression_policy reports both, plus the provider's session-duration caps, so you can see which limit you are actually up against.

Live transcription: a different kind of session¶

A transcription session is not a conversation with the model's voice turned off. There is no model turn at all: you send audio, you get transcripts, and there is nothing to interrupt, no tool to call and no answer to cancel. send_text raises, naming the conversational kind, and tools= is refused at construction by the row's supports_tool_calls flag.

from doc_processing_toolkit.realtime import open_transcription_session

session = await open_transcription_session("gpt-transcribe")
try:
    await session.send_audio(pcm_24k)
    await session.commit_audio()
    async for event in session.events():
        if event.type == "transcript_complete":
            print(event.item_id, event.text)
finally:
    await session.close()

print(session.transcript, session.cost)

Read item_id, not the order events arrive in. Each committed stretch of audio is transcribed independently, so a later utterance can finish before an earlier one — session.transcripts is keyed by item id for exactly that reason, and session.transcript joins them in completion order. Closing the session drains whatever is still being transcribed first (bounded by DEFAULT_FINAL_DRAIN_SECONDS, 2 s, and by "no items still open"), because a turn you have already paid for is often still in flight when you stop reading; a drain that cannot finish warns and names the lost item ids.

Row	Provider	Billing	Notes
`gpt-transcribe`	OpenAI	$0.0045 / minute	The same row you use for a recorded file, on a second surface at the same price. Its `transcription_input_mode` is `committed`, so with turn detection off you call `commit_audio()`
`gpt-live-transcribe`	OpenAI	$0.017 / minute	Socket-only, `continuous`
`gpt-realtime-whisper`	OpenAI	$0.017 / minute	Socket-only, `continuous`; keyword biasing is a `gpt-transcribe`-family option and this row declares `supports_keywords: false`
`gemini-3.5-transcribe-live`	Google	tokens: $3.50/M audio in, $21.00/M text out	Plain captions, a ten-minute cap it cannot resume from, and an estimated bill — see below

The three OpenAI rows are billed by duration, and the toolkit bills the seconds the provider itself reported (session.provider_audio_seconds, summed from each completed item's usage.type == "duration") rather than the time your socket happened to be open; session.billed_audio_seconds is the figure actually used, falling back to connected wall clock — and a session billed on that fallback sets session.cost_estimated, because the socket's clock is not what the provider charged for (it used to be reported as exact). A duration-billed session can never report a confident $0.00 — an unmeasurable one bills the MINIMUM_BILLABLE_MINUTES floor and sets session.cost_estimated.

The OpenAI rows take the five guidance fields of the provider's transcription object, as keywords to open_transcription_session(...) or to configure(...): language (an ISO-639-1 code), languages (candidate codes when the language is not known in advance), prompt (free text steering style or continuing a previous segment), keywords (words or phrases to bias towards) and delay (latency traded for accuracy, one of realtime.TRANSCRIPTION_DELAY_LEVELS: "minimal", "low", "medium", "high", "xhigh"). They used to be unreachable — passing one raised a bare TypeError. The toolkit checks their shape only — a string where one is expected, a non-empty list of strings for languages and keywords, a known delay — and raises RealtimeValidationError otherwise; which model honours which field is the vendor's call, so a row's supports_keywords: false is not enforced on this socket. noise_reduction ("near_field", "far_field", or False for off) is available here too.

session = await open_transcription_session(
    "gpt-transcribe", language="en", keywords=["doc-processing-toolkit"]
)

The connect URL is registry data rather than a branch on the session type: realtime_connect_query and realtime_model_in_url produce ?intent=transcription with no model= on those rows, because the model is named inside session.update. Absent, they reproduce the conversational URL byte for byte.

Gemini live captions behave differently¶

gemini-3.5-transcribe-live is a captions-only Live socket, and three things about the live protocol are worth knowing before you build on it.

Interim captions are cumulative snapshots, not fragments. While audio streams the server sends the whole utterance so far ("This", "This is a", "This is a test"), so each neutral transcript_delta carries only what the latest snapshot added. The completed caption is the authoritative text.
The settled caption arrives only after the end of the audio stretch is marked. commit_audio() sends audio_stream_end, and close() sends it for audio you never flushed. Call commit_audio() at each utterance boundary, or the caption you are waiting for never settles. Under manual turn control (turn_detection={"disabled": True}) the same call sends activity_end, closing the activity_start the first send_audio() of the stretch opened. Google labels no conversation items, so completed captions are keyed by arrival order (caption-0, …) and item_id stays None on a delta.
The provider reports no usage for this model at all. A token-priced session with no token report would close at a confident $0.00, so the adapter estimates instead: input audio tokens from the seconds of audio it sent at the row's input_audio_tokens_per_second (25, registry data), output tokens from the caption text. session.usage_estimated and session.cost_estimated are both True and one WARNING says so. A real usage report, should one ever arrive, replaces the estimate.

Language hints, custom vocabulary and mode. Google documents three options for live transcription, and they are named parameters of open_transcription_session(...) and of configure(...) on this row:

languages: BCP-47 hints, sent as language_codes. An empty list asks for automatic identification explicitly.
custom_vocabulary: up to 1,000 phrases. Google says results are best with up to 100, and the toolkit warns above that.
mode: "VERBATIM", the default, or "SMART", which removes disfluencies and formats the text.

Each option is gated on a registry key: supports_language_hints, supports_custom_vocabulary (with max_custom_vocabulary_terms and recommended_custom_vocabulary_terms), and supported_transcription_modes. They are validated by the same rules as file transcription. A row without the key refuses the parameter by name. Before this release languages= raised a bare TypeError, although the row said supports_language_hints: true.

session = await open_transcription_session(
    "gemini-3.5-transcribe-live",
    languages=["en-US"],
    custom_vocabulary=["doc-processing-toolkit", "Gemini Live"],
    mode="SMART",
)

Diarization and word timestamps are refused locally: the row publishes neither, and the API would accept the option and return plain captions — a silent wrong answer rather than an error. Use transcribe_audio on a completed recording when you need annotations.

The ten-minute cap ends the stream; it does not reconnect

This row publishes no session resumption, so a reconnect would be an unrelated session with no memory of the audio already transcribed and the transcript would silently restart mid-sentence. The adapter therefore ends the stream with an error event naming the cap. session.compression_policy["limits"]["max_session_minutes"] tells you the figure before you hit it; for a long meeting, open a fresh session before the cap and join the transcripts yourself. Every conversational Gemini Live row declares resumption, so they are unaffected.

Live translation: the model as interpreter¶

A translation session is the one realtime session where the model is not talking to you. It listens to a speaker and interprets: it does not answer questions, follow instructions, call tools or take turns. Both vendors are explicit about the difference — Google contrasts "the model acts as an assistant" with "the model acts as an interpreter … translates as the speaker talks without waiting for turns", and OpenAI contrasts "uses a conversation and response lifecycle" with "streams continuously from incoming audio … you don't call response.create". That is why it is its own session kind rather than a system prompt.

from doc_processing_toolkit.realtime import open_translation_session

session = await open_translation_session(
    "gemini-3.5-live-translate-preview", target_language="en"
)
consumer = asyncio.create_task(drain_events(session))   # audio_delta → speaker
try:
    for chunk in japanese_microphone:                   # 16 kHz PCM16, paced
        await session.send_audio(chunk)
finally:
    await session.close()
print(session.source_transcript, "->", session.translated_transcript)

open_translation_session(model, target_language=...) is open_realtime_session(model, session_kind="translation", ...) under its own name, and target_language is required. Google documents a default of "en"; the toolkit refuses instead, because a session that silently interpreted into a language nobody asked for is a wrong answer billed by the minute. Everything a conversation has and an interpreter does not — voice, instructions, output_format, modalities, turn_detection, thinking_level, proactive_audio, enable_affective_dialog, context_compression, tool_call_behavior, tool_response_scheduling — is refused locally rather than accepted by the provider's config object and ignored, and send_text raises naming send_audio.

Keep sending audio, including the silence between phrases. Both transcripts — and Gemini's usage reports — advance only while input audio keeps arriving, so a caller that streams a recorded clip and then stops sending loses the tail of both transcripts (the translated audio still completes). Follow a clip with a second or two of paced silence, exactly as a microphone would, before you close. There are no turns to detect, and OpenAI documents that audio resumed after a gap is treated as contiguous with what came before rather than as a real-world pause.

Both transcripts come back, and the role tells them apart. role="user" is what the speaker was heard to say, in the language they said it in; role="assistant" is the translation. Where a provider labels the language it detected, event.language carries the BCP-47 code and session.detected_source_languages accumulates them in first-seen order — Gemini does this, OpenAI labels nothing. You cannot set a source language on either provider, so source_language= is accepted and ignored with one warning rather than quietly doing nothing.

One target language per session. That is a vendor constraint, not a toolkit one: Gemini's config carries a single target_language_code and OpenAI's guide says outright to create one session per output language. The code is checked against the model's own published list before the socket opens — 79 codes on Gemini, 13 on OpenAI — case-insensitively and in the vendor's own spelling, so "PT-br" goes out as pt-BR and "zh" is refused with a "did you mean zh-Hans / zh-Hant" that reflects how Google actually splits Chinese.

target_languages= and echo_target_language=. target_languages=[code] with one element is accepted as a synonym for target_language=; more than one code is refused, because neither row publishes supports_multi_target_translation, and a bare string is refused with a pointer to the singular form. echo_target_language=True asks the interpreter to repeat speech that is already in the target language instead of staying silent for it; it is Google-only, gated on the row's supports_echo_target_language (true on gemini-3.5-live-translate-preview, false on gpt-realtime-translate), and refused elsewhere.

	Gemini live translate	OpenAI translate
Row	`gemini-3.5-live-translate-preview` (preview)	`gpt-realtime-translate`
Transport	the same Gemini Live SDK socket, plus a `translation_config`	a raw WebSocket to `/v1/realtime/translations`
Audio	16 kHz PCM16 in, 24 kHz out	24 kHz PCM16 both ways; the vendor recommends 200 ms chunks (`OPENAI_TRANSLATION_FRAME_MS`) and `send_audio` forwards yours as they are
Billing	tokens ($3.50/M audio in, $21.00/M audio out)	duration ($0.034 per minute), always `cost_estimated`
Source transcripts	on by default	opt-in; engages a second, separately priced model
Detected language	reported on each transcript	not reported

The two billing rows differ in what the provider tells you, not just in the tariff. Gemini does report usage — one usage_metadata per second of input audio, 25 audio tokens each way — but the first one lands three to four seconds into the stream, so a shorter session closes on an audio-derived estimate instead, with session.usage_estimated set. OpenAI sends no usage or duration event of any kind on that socket, so the toolkit bills the audio it appended and flags every cost from the row. Both cases are spelled out under Sessions billed by the second.

The OpenAI row is the module's one documented raw-WebSocket exception

The installed openai SDK hardcodes /realtime in AsyncRealtime._prepare_url and declares no translation session type, so that endpoint is unreachable through it. The exception is confined to one adapter class and adds no dependency: websockets is already a hard requirement of google-genai, which the toolkit pins. The path itself is registry data (realtime_connect_path), not a literal — a second endpoint on the same host is a different row, not a branch on the session type.

The Gemini row is a preview, and never a default

gemini-3.5-live-translate-preview has no announced shutdown date, publishes no session cap, and declares no session resumption — so the inherited reconnect refuses rather than silently opening an unrelated session mid-sentence. (The live server does send resumable session_resumption_update handles unasked; the row stays supports_session_resumption: false until a resumed interpreter session has actually been exercised.) Treat an entitlement refusal at connect as a normal outcome rather than a bug.

Ending a translation session¶

You decide when it ends, and close() waits for the tail. Neither protocol has a reliable end-of-stream event — Gemini's turnComplete is not what a turn-free stream produces, and OpenAI has no response to complete — so response_complete is emitted only opportunistically and must never be your only exit. Closing the session flushes first (on Gemini by marking the audio stream ended, which is what makes the last utterance settle; on OpenAI by sending session.close and reading until the server answers session.closed) and folds whatever arrives into the transcripts and the usage totals. session.final_drain_events is what the drain saw. Stop iterating events() before you call close() — the drain reads the same socket.

commit_audio() exists on the Gemini translation session only: it marks the audio stream ended without closing the socket, so a settled utterance can be read before the next one starts. The OpenAI translation session has no commit — close() is its only flush.

The drain is bounded by a wall clock, so a socket that has gone quiet cannot hang your shutdown. On OpenAI that budget is its own: a live run measured session.closed arriving 3.9 seconds after session.close, behind 34 flushed audio deltas, so the shared 2-second realtime drain closed every session early and dropped that audio. DEFAULT_TRANSLATION_CLOSE_DRAIN_SECONDS is 10 seconds — a budget, not a delay: the drain returns the moment the confirmation arrives — and it is overridable per session:

session = await open_translation_session(
    "gpt-realtime-translate",
    target_language="es",
    provider_options={"close_drain_seconds": 20.0},   # a ceiling, not a wait
)

close_drain_seconds is read at connect and never reaches session.update. The other provider_options this adapter takes are api_key, base_url, websocket_base_url, open_timeout and safety_identifier (OpenAI's OpenAI-Safety-Identifier header).

OpenAI's session.created carries the server's default target, not yours

A live session opened for "en" was created with "es" — the server's own default — and only session.updated reports the language this session actually got. The adapter therefore logs the confirmed target from session.updated, and a confirmed language that differs from the one you asked for is a warning, not a silent substitution.

GPT-Live: a different protocol, not a different model¶

open_live_session("gpt-live-1") opens a session that looks like the voice conversation you already know and works almost nothing like it underneath. GPT-Live is a separate API family on /v1/live/* with its own event vocabulary, and the toolkit surfaces the differences rather than papering over them, because each one changes how you write the loop. It is still SDK-native: everything goes through the openai SDK's own Live client, so there is no raw WebSocket code and no new dependency.

You never ask it to speak. There is no audio commit and no response trigger: you stream audio continuously and GPT-Live decides when to listen and when to talk. commit_audio() and cancel_response() both raise rather than pretending — the model is full duplex and handles interruption itself; to redirect it, use append_instructions(...).

There is no end-of-response event. The if event.type == "response_complete": break idiom that works on gpt-realtime will never fire here, because the protocol emits nothing that marks the end of a spoken answer. Drive your loop from your own playback queue, or from the transcript, and end the session when your application decides the conversation is over. The terminal event is session_closed, and it only arrives after you ask for it.

Transcripts have no item ids. Deltas carry the text, a start time and an end time and nothing else, so event.item_id is None on every one of them and the toolkit does not invent a substitute that would look provider-issued. Group them by speaker and by the start_ms/end_ms timeline; session.input_transcript, session.output_transcript and session.transcript_fragments keep the accumulation for you.

Tools run on a backend, not on the voice model. GPT-Live delegates. By default it delegates to your application (session.delegation.created arrives as a delegation_created event, which you answer with the three context appends). With provider_options={"backend_model": ...} the API manages a Responses backend instead and your tools= become its function tools, which the toolkit still executes locally — including your MCP servers, which is the only way they can reach a Live session, because the delegation surface accepts only function and web_search entries and has no remote-MCP tool type. Without a backend model, tools= is refused rather than silently dead.

from doc_processing_toolkit.realtime_gpt_live import open_live_session

session = await open_live_session(
    "gpt-live-1",
    voice="marin",
    instructions="Be concise. Delegate order questions to the backend.",
    tools=[order_lookup_server],
    provider_options={"backend_model": "gpt-5.6-luna"},
)
try:
    async for chunk in microphone:          # 24 kHz mono PCM16, paced
        await session.send_audio(chunk)
finally:
    await session.close()
print(session.billed_audio_seconds, session.cost, session.cost_excludes)

session.append_instructions(), .append_thinking() and .append_commentary() steer a running conversation (each capped at 500 tokens by the API); .mute_input() / .unmute_input() stop the model hearing you, though not the clock, which runs through silence by design. send_text queues a user message for the backend — the voice frontend has no user-text event at all — so under the default client delegation it raises, naming the three routes that do work: startup history through provider_options={"initial_input": [...]}, the three appends, or configuring a backend model. .session_id, .close_reason and .finalization_confirmed say what happened to the socket: close_reason is one of close_requested, expired, content, remote_hangup or connection_lost, and finalization_confirmed records whether session.closed actually arrived.

Voices. gpt-live-1 publishes 22 voices in supported_voices: the ten Realtime built-ins (alloy, ash, ballad, cedar, coral, echo, marin, sage, shimmer, verse) and the twelve that OpenAI's Live conversations guide adds for GPT-Live (quartz, ripple, vesper, willow, stone, gleam, meridian, bossa, tempo, beacon, delta, cinder). marin is the default. A voice name is now validated locally. An OpenAI custom voice, voice={"id": "voice_..."}, bypasses the list, because the guide says gpt-live-1 supports custom voices. The list and the custom-voice object ship together, so no voice the API accepts becomes newly rejected.

No noise_reduction. The option belongs to the Realtime audio.input object, which GPT-Live does not have, so it is refused by name at configure() time like every other Realtime-only option — the conversational, transcription and translation sessions keep it.

Images go to the backend, never to the voice model. send_image() is refused on a GPT-Live session — the audio frontend accepts none — so route a photo to a vision-capable backend and hand GPT-Live the text it found.

The tool round trip waits for the delegated response to finish

One response.create goes out per delegated response, once its terminal lifecycle event has arrived and every call it made has been answered — the forwarded lifecycle events carry response.output: [] even while function calls still need results, so waiting costs nothing and not waiting strands a second parallel call behind a premature continue. A live run confirmed the order: the result goes out on response.output_item.done, response.completed follows about 0.15 s later, the single response.create follows that, and a second backend response carries the answer. The wait is registry data (live_tool_followup_waits_for_response_end, default true), so a provider change is a JSON edit rather than a code change.

Not in this release: stored sessions and forking, recording download, SIP call control, the sideband monitoring connection, and the WebRTC transport.

Usage and cost¶

Usage is tracked per modality in both directions, including the cached share where the provider reports one, and priced through calculate_cost under the registry's audio_realtime schema — or, on a row billed by the clock rather than by tokens, under per_minute (see Sessions billed by the second). session.usage, session.cost and session.cost_components are live throughout the session and final after close().

session.usage.input_audio_tokens
session.usage.input_image_tokens       # stills and frames
session.usage.output_audio_tokens
session.usage.output_thinking_tokens   # a SUBSET of output_text_tokens
session.usage.cached_input_tokens      # a SUBSET of total_input_tokens
session.usage.as_dict()                # every counter plus derived totals
session.cost_components["duration_usd"]  # a per_minute row's whole charge

session.cost_components is the per-component split of session.cost — the calculate_cost component keys, refreshed with the cost. Every entry but total_usd sums to total_usd on every row: a token-priced session spreads its charge over the input, cached and output components, and a duration-billed one (transcription, gpt-realtime-translate, gpt-live-1) carries all of it in duration_usd. The attribute is new; session.cost itself is unchanged.

Cached counts follow the toolkit-wide convention: they are a subset of their non-cached counterparts, never an addition (see Cost Tracking). So does output_thinking_tokens: Gemini Live reports thinking as thoughts_token_count, it is billed as text-rate output, and the toolkit folds it into output_text_tokens and reports it separately only so you can see how much of the answer was reasoning. Before this release it was never read, so a thinking Live model's reasoning went unbilled — 649 tokens on one probe turn of gemini-3.8-live-extended-thinking.

When an OpenAI response reports a cached total with no per-modality split, the toolkit spreads it across audio, image and text in proportion to that response's input and sets session.cost_estimated, because the split is an estimate. It used to label the whole total text, where the per-modality clamp discarded it, so cached audio was billed at the full audio rate — $0.160 instead of $0.011 for one 5,000-audio-token response on gpt-realtime. A reported split is used as reported and stays exact. The audio_realtime schema prices text, audio and image separately in each direction — input, input_audio, input_image, output, output_audio, plus cached_input / cached_input_audio / cached_input_image where published.

input_cached_image_tokens is clamped per modality rather than against the prompt total, so a provider reporting more cached image tokens than image tokens cannot leak the surplus into the discount. On OpenAI both counts come from input_token_details.image_tokens and cached_tokens_details.image_tokens; on Google the IMAGE and VIDEO modality rows are summed, because a frame is reported under either and both bill at the same input_image rate.

The rows that publish an image rate¶

Row	`input_image`	`cached_input_image`
`gpt-realtime-2`, `gpt-realtime-2.1`, `gpt-realtime-1.5`	5.00	0.50
`gpt-realtime-2.1-mini`	0.80	0.08
`gemini-3.1-flash-live-preview`, `gemini-3.8-live`, `gemini-3.8-live-extended-thinking`	1.00	— (no Live row publishes a cached rate)
`gpt-realtime`	5.00	0.50

gpt-realtime accepts image input too — OpenAI's model card lists text, audio and image input and the pricing page publishes the same $5.00 / $0.50 pair — so every conversational OpenAI row now carries one. Until this row gained the pair, an image token on gpt-realtime billed at the text input rate ($4.00/M) and now bills at $5.00/M. On every other row the three-way split left audio-and-text costs unchanged to the last cent, because no row had published both an audio and an image rate before it.

Session duration is wall-clock, not frame count¶

session.session_duration_seconds accumulates the time the transport was actually connected, summed across reconnects. A realtime provider that bills per minute bills connection time; a frame-derived figure would need a per-provider sample-rate assumption and would under-count the silence the provider still charges for. Gemini Live's hidden session-duration cap closes one billing segment and opens another, so the reconnect gap is excluded.

The figure is passed to the cost engine as audio_minutes, and the zero-token early return that used to short-circuit pricing no longer fires for a duration-billed row. Every conversational realtime row is token-priced under audio_realtime, as is gemini-3.5-live-translate-preview; the per_minute rows are the three OpenAI transcription rows, gpt-realtime-translate and gpt-live-1. On all of them a provider-reported duration beats the wall clock where the provider reports one — see Live transcription and Sessions billed by the second.

Gemini Live has no published cached rate

Google's pricing page publishes no cached-input rate for Live models, so the registry sets none and the gap is recorded in pricing._comments. Cached audio on gpt-realtime is priced.

Two Gemini accounting behaviors were settled against the live API rather than assumed, and both matter if you are reconciling a bill:

Gemini Live sends exactly one usageMetadata per turn carrying that turn's usage, not a running total — responseTokenCount was observed decreasing across turns (25, 27, 23). Reports are therefore summed, not diffed.
promptTokenCount includes tokens that appear in no modality row (the system instruction), so the audio share is taken from the details and the remainder bills as text.

Exporting a session and seeding the next one¶

session.export() returns the session's record as JSON-clean data, at any time and final after close():

record = session.export()
# {"schema_version": 1, "provider": "openai", "model": "gpt-realtime", "session_kind": "realtime",
#  "started_at": "2026-09-24T10:00:00+00:00", "closed_at": "...", "close_reason": "closed",  # "error" after a transport failure
#  "duration_seconds": 12.4, "usage": {...}, "cost": 0.0123, "cost_estimated": False, "cost_components": {...},
#  "transcript": [{"role": "user", "text": "...", "item_id": None, "completed_at": "..."}, {"role": "assistant", ...}]}

transcript is the text of every completed turn, in completion order: the turns you sent with send_text, the caller's speech when input_transcription is on (tagged user), and the model's own transcript (assistant); an answer cut short by a barge-in is kept and marked "interrupted": True. Audio is never retained, so an hour of conversation is a few KB. The same list is live on session.conversation. A transcription session exports its completed captions the same way. A translation session and GPT-Live retain no transcript by design (an interpreter's two transcripts are on source_transcript / translated_transcript and its events; GPT-Live's text is on its own event stream) and export usage and cost with an empty list. Every family's close() sets closed_at and close_reason; on GPT-Live close_reason is the provider's session.closed reason when one arrived.

Seeding the next session. provider_options={"initial_items": [...]} on an OpenAI Realtime or Gemini Live session takes the exported entries, {"role": "user" | "assistant", "text": ...} (content is accepted in place of text, which is GPT-Live's initial_input shape), and hands them to the model before the first response: OpenAI as completed message items right after session.update, Gemini as prior turns through send_client_content(turns=..., turn_complete=False) at connect, and again after a reconnect that could not resume. No response is requested for them and they are not part of the new session's own transcript. The option is validated at construction: a non-list, an unknown role or an empty text raises RealtimeValidationError, and so does giving it to a transcription, translation or GPT-Live session (GPT-Live's route is provider_options={"initial_input": ...}; a transcription session's prompt takes the tail of a transcript instead).

record = session.export()
seed = [{"role": e["role"], "text": e["text"]} for e in record["transcript"]]
resumed = await open_realtime_session("gpt-realtime", provider_options={"initial_items": seed})
await resumed.send_text("Where were we?")

Provider notes¶

OpenAI¶

OpenAIRealtimeAdapter drives AsyncOpenAI().realtime.connect(model=...) — the GA, non-beta surface, so the SDK owns the URL, auth header, and handshake and no OpenAI-Beta header is hand-set.

Output formats are the GA names: audio/pcm, audio/pcmu, audio/pcma. The retired beta names (pcm16, g711_ulaw, g711_alaw) are rejected outright by a live session. audio/pcm also needs an explicit rate, which the adapter sources from the registry.

The socket's protocol family and the model used for input transcription are registry values too (realtime_session_type, default "realtime", and default_input_transcription_model), not literals in the adapter. All five conversational OpenAI rows now name gpt-transcribe as the input-transcription model. They used to name whisper-1, which OpenAI removes on 2027-02-26 together with gpt-4o-transcribe, gpt-4o-mini-transcribe and gpt-4o-transcribe-diarize. gpt-transcribe is the cheapest non-retiring model the Realtime reference accepts for audio.input.transcription.model, at $0.0045 a minute against whisper-1's $0.006. So input_transcription=True puts a different model on the wire from this release. To keep the old one until its retirement, pass input_transcription={"model": "whisper-1"}. The whisper-1 row stays registered. The transcription rows set realtime_session_type: "transcription" and name themselves as their own transcription model, which is what makes their socket a different protocol family rather than a branch in the adapter. A row that requests transcription without publishing a default fails locally, naming the registry path.

audio.input options. Three named options map onto the audio.input object:

noise_reduction: "near_field" for headsets, "far_field" for laptop and conference-room microphones, or False to switch it off. False is sent as null. The conversational, transcription and translation sessions share one vocabulary, realtime.NOISE_REDUCTION_TYPES; GPT-Live has no such field and refuses the option by name.
input_transcription: also takes a dict of transcription guidance on a conversational session, {"model", "language", "languages", "prompt", "keywords", "delay"}. The dict is merged over the registry model, and a "model" key replaces that model. language and languages are alternatives: OpenAI says never to send both, so supplying both is refused. The accepted keys are realtime.INPUT_TRANSCRIPTION_KEYS.
A passthrough provider_options={"audio": {"input": {"transcription": {...}}}} is deep-merged into that object. It used to replace the object wholesale, which silently dropped the registry's transcription model.

session = await open_realtime_session(
    "gpt-realtime-2.1-mini",
    noise_reduction="far_field",
    input_transcription={"language": "de", "keywords": ["IBAN"]},
    thinking_level="low",
)

Reasoning effort. gpt-realtime-2, gpt-realtime-2.1 and gpt-realtime-2.1-mini publish supported_thinking_levels: minimal, low, medium, high and xhigh. On those rows thinking_level= is sent as session.reasoning.effort. It used to be dropped with a warning. It still is on gpt-realtime and gpt-realtime-1.5, which are not reasoning models. OpenAI's voice-prompting guide recommends starting at low. Higher effort can add latency and output tokens, so the change has a kill switch: DPT_OPENAI_REALTIME_REASONING=off restores the drop (see Configuration).

Custom voices. voice= also takes an OpenAI custom-voice object, {"id": "voice_..."}, on a conversational session, on GPT-Live and in synthesize_speech. The object bypasses supported_voices, since a custom voice is by definition not on that list, so only its shape is checked. Whether your account may use the voice is for the API to decide: OpenAI limits custom voices to eligible customers. Gemini Live refuses the object by name. In synthesize_speech the Gemini 3.8 TTS rows accept one as well (see Multimedia).

A raw provider_options["tools"] replaces the dispatched function tools in session.update. It is still sent, with one WARNING.

xAI escape hatch

xAI's voice API is deliberately OpenAI-Realtime compatible, so the adapter accepts base_url, websocket_base_url, and api_key overrides through provider_options:

session = await open_realtime_session(
    "gpt-realtime",
    provider_options={"base_url": "https://api.x.ai/v1", "api_key": XAI_KEY},
)

Overriding only the endpoint works too — with api_key omitted the client falls back to OPENAI_API_KEY, read from the environment or the repo-root .env.

Google¶

GeminiLiveAdapter drives client.aio.live.connect(model=..., config=...), so the SDK owns the BidiGenerateContent handshake and framing.

The session-duration cap is invisible to callers. Gemini Live caps sessions server-side (15 minutes, 2 with video). The adapter enables session_resumption at connect, records every resumption handle the server issues, reacts to the server's goAway warning, and reconnects with the stored handle — replaying the same configuration. You see one uninterrupted events() stream with a single session_resumed event marking the seam.

If the connection drops before the server has issued any handle, the adapter still reconnects, but there is nothing to replay and the new session has no memory of the conversation. That seam is marked by session_restarted (EVENT_SESSION_RESTARTED) and a WARNING instead of session_resumed: treat it as a context reset and resend whatever the next turn needs. session_resumed now means a handle was sent. The transcription and live-translation adapters inherit both events.

Reconnects use exponential backoff (capped at 8 s) and are bounded by max_reconnects in provider_options, default 10. Exhausting the budget logs an error and ends the stream rather than retrying forever.

A text turn takes a different message per model. send_text used to call send_client_content everywhere, which Google documents on gemini-3.1-flash-live-preview as seeding initial context only — so every text turn after the first was silently dropped on that model. The route is now driven by the registry key supports_client_content_mid_session: 3.1 Live uses send_realtime_input(text=...), the 3.8 Live rows keep send_client_content. No name matching is involved.

provider_options also accepts any LiveConnectConfig field verbatim — temperature, context_window_compression, proactivity, and so on — and the capability gates apply to it too, so a feature the model does not have fails locally rather than at the API. Pass tools through open_realtime_session(tools=...) rather than as a raw tools config: only the former is dispatched, hooked and bounded.

Six passthrough keys replace what the adapter builds

provider_options is applied to LiveConnectConfig with a flat update, so a key the adapter also builds replaces the adapter's object wholesale. Each one below is still sent, with one WARNING per key per session naming what it overrides (the map is GeminiLiveAdapter.passthrough_overrides):

Passthrough key	What it replaces
`session_resumption`	The adapter-managed transparent reconnect
`speech_config`	`voice=`
`thinking_config`	`thinking_level=`
`realtime_input_config`	`turn_detection=`, including manual turn control
`input_audio_transcription`	`input_transcription=` (on the captions row: `languages=`, `custom_vocabulary=`, `mode=`)
`tools`	The dispatched function tools

The guard applies to all three Gemini adapters: conversational, captions and translation. It warns and does not refuse. A refusing variant would be a behaviour change and is a known limitation.

Tool schemas are normalised for Gemini Live. The realtime path now runs the same $ref / $defs inlining and keyword stripping the text loop uses, so a tool whose arguments are a Pydantic model no longer fails at connect time while the identical tool works in an Agent run. It additionally filters the JSON Schema meta-keywords — $schema above all — that google.genai.types.Schema rejects outright, checking each node against the installed SDK's own field list so a field a future google-genai adds stops being stripped automatically.

The ordinary Google tool path applies the same filter

Since this release the ordinary Google tool path applies the same filter (utils._strip_non_google_schema_keys, shared with this adapter), so an MCP tool schema carrying $schema no longer raises while its FunctionDeclaration is built. See Providers.

No raw WebSocket code, no new dependency¶

Every adapter goes through its vendor SDK's own client — the conversational and transcription sessions, Gemini live translation, and GPT-Live, whose whole /v1/live/* surface ships in the installed openai SDK. There is exactly one exception, and it is documented rather than incidental: gpt-realtime-translate lives on /v1/realtime/translations, which that SDK cannot address at all (it hardcodes /realtime and declares no translation session type), so its adapter speaks a raw WebSocket. The websockets package every one of them uses underneath is already a hard requirement of google-genai, which the toolkit pins, so realtime still adds no dependency.

Realtime is deliberately not an MCP tool¶

The multimedia MCP server exposes TTS, STT, and image generation, but not realtime. MCP tool calls are request/response; a realtime session's value is in sub-second audio frames flowing both ways, interruption handling, and a connection held open across many turns. None of that survives being squeezed through one-shot tool calls, so realtime stays a Python API.

Errors¶

Exception	Raised when
`RealtimeValidationError`	The model is not a registered realtime model, or an option it does not support was passed
`RealtimeConnectionError`	No client could be built for the provider (missing SDK or API key)

Provider-reported errors mid-session arrive as error events rather than exceptions, so a consumer loop decides whether to continue or break.

Realtime Voice Session recipe — a runnable walkthrough: one turn, tools, a picture, captions, an interpreter and GPT-Live with a reasoning backend
Multimedia — one-shot TTS, STT, and image generation/editing
Client Integration — what an application around this API has to own
Cost Tracking — pricing schemas and the cached-tokens convention
MCP — the multimedia tool server, and why realtime is not on it
Streaming — token streaming for ordinary text agent runs
Providers — provider routing and the model registry