Recipe: Built-in Tools (Web Search / Code Execution)¶

Goal¶

Enable provider built-in tools and combine them with normal prompting or schemas.

Example¶

from doc_processing_toolkit import Agent
from doc_processing_toolkit.schemas import BuiltInToolConfig

agent = Agent(
    model="gpt-4o",
    instructions="You are a research assistant.",
    built_in_tools={
        "web_search": BuiltInToolConfig(enabled=True),
        "code_execution": BuiltInToolConfig(enabled=True),
    },
)

result = await agent.run("Research Fibonacci sequence and implement it in Python.")
print(result.output)

Tool names across providers¶

Built-in tools are requested by a standardized name whatever the provider. Two of them have a different name on Google's API, and the toolkit rewrites those for you:

You pass	Anthropic	OpenAI	Google	Fireworks
`web_search`	`web_search`	`web_search`	`google_search_grounding`	—
`url_context`	—	—	`google_url_context`	—
`web_fetch`	`web_fetch`	—	—	—
`code_execution`	`code_execution`	`code_execution`	`code_execution`	—
`context_editing`	`context_editing`	—	—	—
`remote_mcp`	`remote_mcp`	`remote_mcp`	—	`remote_mcp`
`tool_search`	`tool_search`	`tool_search`	—	—

Only the first two rows are aliases — web_search and google_search_grounding are the same tool under two names, so passing either works and the toolkit sends the one the provider expects. The rest are separate tools that simply do not exist everywhere.

That distinction matters most for the two that look interchangeable and are not. web_fetch (Anthropic) retrieves a URL you name and hands the model its content; url_context (Google) grounds a prompt in a URL you name. They answer similar questions with different mechanics and different citation behaviour, so the toolkit will never substitute one for the other silently. What it does instead is say so — ask for web_fetch on a Gemini model and the tool is dropped with:

Tool 'web_fetch' is not supported by model 'gemini-3.7-flash'.
Supported tools: ['code_execution', 'google_url_context', 'google_search_grounding']
Did you mean 'url_context'?

The suggestion is checked against that model's own row before it is offered, so it never points at a second tool the model also lacks — an image row advertising only search grounding gets no suggestion at all, and neither does a provider with no counterpart (Fireworks, which advertises only remote_mcp). A name copied across providers is understood too: google_url_context handed to Anthropic is answered with web_fetch. The message is a logger WARNING on the run; it is also the third element of utils.validate_built_in_tools(...), if you want to surface the hint in your own UI rather than read it out of a log. utils.nearest_built_in_tool(tool_name, provider, supported) answers the same question on its own and returns None when there is nothing honest to suggest.

Tool search and deferred tools¶

A model offered fifty tools pays for fifty tool schemas in every request, and on Anthropic those schemas sit inside the cached system block — so a tool surface that changes between turns silently destroys the cache read for the whole run. Tool search is the providers' answer: the tools you defer are sent with defer_loading: true, which keeps their schemas out of the prompt until the model searches for them by name.

from doc_processing_toolkit import Agent
from doc_processing_toolkit.schemas import BuiltInToolConfig

agent = Agent(
    model="claude-haiku-4-5",
    tools=[{"type": "mcp_script", "path": "catalog_server.py", "name": "catalog"}],
    built_in_tools={"tool_search": BuiltInToolConfig(enabled=True)},
    defer_tools=lambda tool_name, server_name: server_name == "catalog",
)

result = await agent.run("Look up SKU 4417.")
print(result.deferred_tools)   # the tools that were actually hidden

defer_tools takes either a set of tool names or a predicate called as predicate(tool_name, server_name), where server_name is the MCP server the tool came from (None for a tool with no server). A one-argument callable is a TypeError at construction. YAML takes the list form (defer_tools:), because a predicate is only expressible in Python, and the agent factory's create_agent / create_agent_session take enable_tool_search, tool_search_variant and a defer_tools list.

On Anthropic, config={"variant": "bm25"} swaps the regex search tool for the BM25 one — regex is the default and an unknown variant raises locally, naming both. On OpenAI the hosted search tool has no variants, so a variant there is ignored with a warning and never reaches the wire; that is deliberate, so one configuration survives a fallback chain that crosses providers.

Nothing here can make a tool unreachable

If tool search is not enabled, or the model's registry row does not list tool_search, or the provider has no native tool search at all (Google and Fireworks), every tool loads eagerly, one warning names why, and the run proceeds. AgentResult.deferred_tools is empty in each of those cases, which is the signal to go looking for the warning. The tool-search tool itself, every other built-in tool, and the tool the toolkit adds for tool-based structured output are never deferred — eligibility is structural, so no predicate can defer them.

Tool search is not separately metered on either provider: its built_in_tool_pricing rows are explicitly empty, and a run that searches costs the same as the same run without it. A search that fails is logged at WARNING with its error code and the run continues with the tools already loaded. A server_tool_use block named tool_search_tool_regex or tool_search_tool_bm25 is reported back as plain tool_search in built_in_tools_invoked, rather than as a wire name no caller can configure.

Which models. Fourteen Anthropic rows and thirteen OpenAI rows (gpt-5.4 and later) carry it today. Ask the registry rather than a list: get_model_info(model)["supported_built_in_tools"] contains tool_search exactly where it is available. One Anthropic row carries it on live evidence rather than on the published compatibility table, which omits it: claude-sonnet-5. claude-opus-5-5 was added the same way, verified live on 2026-09-23 with both the regex and bm25 variants before the table named it; the table read on 2026-10-02 lists it, and lists claude-sonnet-5-5, which passed the same live check that day, as did claude-haiku-5-5 on 2026-10-09. Each of the four records its provenance in a _tool_search_note.

Tool search and output_schema do not compose

No source on either provider states that tool search works alongside a response schema, so tool_search is deliberately absent from every row's supports_built_in_tools_and_structured_outputs. Combining them drops the tool with the existing "cannot be used concurrently" warning rather than sending a request that would 400.

See Prompt Caching for what deferral does to the cached prefix, and Remote MCP Servers for deferring a whole provider-connected server.

Did the tool actually run?¶

Two result fields look similar and mean different things:

Field	Meaning
`built_in_tools_used`	The built-ins configured on the request. A config echo — it says a tool was offered to the model, never that the model reached for it.
`built_in_tools_invoked`	The subset with observed evidence that it ran this turn. Empty when a configured tool was never invoked.

result = await agent.run("What is 2 + 2?")

result.built_in_tools_used      # ['web_search', 'code_execution'] — both were enabled
result.built_in_tools_invoked   # [] — the model just answered

Do not use built_in_tools_used to decide whether a search happened, whether to bill something, or whether a fallback should fire. It reports the same list on a run that searched ten times and on a run that searched zero times. built_in_tools_invoked is derived separately at return time, so the configured list keeps its exact meaning and continues to gate uploads, MCP wiring, and pricing branches internally.

Evidence differs per provider, because each reports invocation differently:

Provider	What counts as evidence
Anthropic	`server_tool_use` / `mcp_tool_use` blocks in the response, a code-execution container that demonstrably ran, and non-zero search / fetch counters
OpenAI	`search_count > 0` for `web_search`; observed containers or created files for `code_execution`; `mcp_call` output items for `remote_mcp`
Google	`grounding_metadata` for `google_search_grounding`, `url_context_metadata` for `google_url_context`, an `executable_code` / `code_execution_result` part for `code_execution`
Fireworks	An MCP tool-invocation output item for `remote_mcp` — the only built-in Fireworks exposes

The honest per-tool counters (search_count, code_sessions, distinct_urls) remain the finest signal when you need magnitude rather than a yes/no.

Fireworks built_in_tools_used reports custom/MCP tool names

Fireworks populates the field from a placeholder key that carries the MCP/custom tool names the run actually called — and only when built_in_tools was supplied at all. So on Fireworks the field is neither a built-in list nor a config echo. The quirk is preserved deliberately: built_in_tools_used was not redefined on any provider. Read built_in_tools_invoked there, which reports ['remote_mcp'] when a remote MCP tool was genuinely invoked.

Structured output + built-in tools¶

from pydantic import BaseModel
from doc_processing_toolkit import Agent
from doc_processing_toolkit.schemas import BuiltInToolConfig

class ResearchSummary(BaseModel):
    topic: str
    summary: str

agent = Agent(
    model="claude-sonnet-4-5",
    instructions="Use tools and respond with structured output.",
    built_in_tools={"web_search": BuiltInToolConfig(enabled=True)},
)

result = await agent.run("Find current Python release highlights.", output_schema=ResearchSummary)
print(result.output)

Code execution-focused example:

agent = Agent(
    model="gpt-4o-mini",
    instructions="Use code execution for verification.",
    built_in_tools={"code_execution": BuiltInToolConfig(enabled=True)},
)
result = await agent.run("Compute prime numbers up to 30 and summarize the method.")

OpenAI: distinct_urls comes from the searches, not only the citations

A JSON or structured answer carries no url_citation annotations, so reading URLs from the annotations alone left distinct_urls empty after a search that really ran. Every OpenAI request that carries web_search now also sends include=["web_search_call.action.sources"] (added to any include already on the request, never replacing it), and distinct_urls is the de-duplicated union of the cited URLs and the URLs each web_search_call searched, cited ones first, with utm_source stripped. A prose answer can therefore list more URLs than it cites. DPT_OPENAI_WEB_SEARCH_SOURCES=off restores the previous request (no include entry) and the citations-only list.

OpenAI: code interpreter is billed per container actually used

OpenAI charges per code-interpreter container session. The toolkit counts sessions from the container_ids observed in code_interpreter_call output items after the response arrives, rather than counting one session per configured tool. An agent that enables code_execution but whose model never runs code is therefore billed $0 for the tool.

OpenAI: files the code interpreter writes are downloaded (fixed in 0.5.2)

A file the model saves in the container is cited in its reply (a container_file_citation annotation), and the toolkit downloads each cited file to code_execution_output_dir (or, by default, to a fresh ./openai_code_execution_files_<timestamp>_<random>/ per run — see Output files) and lists it in result.files_created. Until 0.5.2 no file was ever downloaded on OpenAI: the download called the SDK with its two ids in the wrong order, every entry of downloaded_files reported success: False with a TypeError in the log, and files_created stayed empty. Both paths are fixed and were checked live: a run with the tool alone, and a run with local tools as well (the tool loop, which now downloads through its async client). The cited name is reduced to a single safe path component before it is written (utils.safe_output_filename, on every provider), so a name such as ../../outside.txt cannot leave the folder; downloaded_files[i]["filename"] is the name written.

Dual built-in tools example:

agent = Agent(
    model="claude-sonnet-4-5",
    instructions="Research and verify with code when useful.",
    built_in_tools={
        "web_search": BuiltInToolConfig(enabled=True),
        "code_execution": BuiltInToolConfig(enabled=True),
    },
)
result = await agent.run("Find and verify the first 10 Fibonacci numbers.")

Anthropic: built-in tools alongside custom tools

On Anthropic, built-in (server) tools execute server-side, and the resulting server_tool_use / *_tool_result block pairs live inside the assistant message. When a turn also triggers custom tools (or pause_turn / max_tokens continuation), the toolkit echoes both halves of every pair back to the API automatically — Anthropic rejects an unpaired server_tool_use with a 400 — so built-in and custom tools can be freely combined in the same run.

Google: built-in tools alongside custom tools (Preview)¶

Gemini 3 text models can now run Google's server-side built-in tools in the same request as your custom @tool functions and MCP tools. This was previously rejected outright — the toolkit raised a ValueError whenever built_in_tools and custom tools were both supplied to a Google model.

from doc_processing_toolkit import Agent, tool
from doc_processing_toolkit.schemas import BuiltInToolConfig

@tool
def lookup_internal_price(sku: str) -> float:
    """Return the internal list price for a SKU."""
    return {"A-100": 42.50}.get(sku, 0.0)

agent = Agent(
    model="gemini-3.5-flash",
    instructions="Research publicly, then price internally.",
    tools=[lookup_internal_price],
    built_in_tools={
        "web_search": BuiltInToolConfig(enabled=True),
        "code_execution": BuiltInToolConfig(enabled=True),
    },
    code_execution_output_dir="./gemini_artifacts",
)

result = await agent.run("Find the market price for SKU A-100 and compare it to our list price.")
print(result.built_in_tools_used)      # ['google_search_grounding', 'code_execution'] — what was CONFIGURED
print(result.built_in_tools_invoked)   # the subset with evidence it actually RAN this turn

built_in_tools_used is a configuration echo

A run where the model answered without searching still reports web_search there. Under combination that distinction matters most — see Did the tool actually run? for the evidence each provider exposes.

Which models¶

Combination is gated on the registry flag supports_concurrent_built_in_and_custom_tools, which is true on every Gemini 3 text row and false on the Gemini 3 image rows and on all Gemini 2.5 models, which still raise a ValueError when both tool kinds are supplied. The flag is the source of truth and the set grows with each Gemini release — read model_information.json, or call get_model_info(<model>), rather than relying on a list here.

The same rows also carry supports_built_in_tools_and_structured_outputs, so they take the native structured-output path (Path A: response_schema in the generation config) instead of the submit_final_response tool fallback. Both keys are absent-means-no, and a test derives the row set from the registry so a new Gemini 3 text row that omits either one fails the gate rather than silently rejecting a feature all its siblings have.

What the toolkit does differently under combination¶

The Gemini API requires a specific request shape when both tool kinds are active, and the toolkit applies it automatically:

Your custom function declarations and the built-ins go into separate Tool objects — one carrying function_declarations, another carrying the built-in configuration.
tool_config.include_server_side_tool_invocations=True is set, so the model's server-side tool calls are circulated back through the conversation as tool_call / tool_response parts. The API's own 400 names this flag when it is missing.
Function-calling mode is left as you set it. AUTO works in this configuration, so tool_choice is passed through unchanged.

Changed in 0.3.0 — the required construction moved, and the old one fails silently

Earlier releases merged both tool kinds into one Tool object and forced VALIDATED mode, which is what the API required when combination first shipped. Google changed the contract: the merged shape is still accepted, but the built-in tool never executes — no error, no warning, just a run where only your custom tools ran. If you pin an older doc-processing-toolkit, combined Gemini 3 runs will look successful while silently skipping the built-in. Verified live on gemini-3.7-flash and gemini-3.5-flash-lite (2026-08-23).

Supported built-ins under combination are web_search (google_search_grounding), code_execution, and url_context (google_url_context). Anything else is dropped with a warning.

Billing follows the same accounting as the non-combination path: grounded searches are counted per iteration from each response's grounding_metadata.web_search_queries, and code execution is billed as one session per run.

Preview caveat: file artifacts under combination are unproven

This limitation was first recorded while the merged-Tool construction meant the built-in never actually ran, so no executable_code or inline_data parts could come back and result.files_created was always empty. Under the split-Tool construction those parts do circulate through the conversation history, and extraction runs on every iteration rather than only on the final response — so artifacts should now surface. That has not been re-verified live since the fix, so treat result.files_created as unproven under combination. If you need the generated files with certainty today, run code execution without custom tools via the structured-output path, where extraction is verified.

Where artifacts land¶

code_execution_output_dir sets the directory for any built-in tool file output. When it is not supplied, the toolkit creates a fresh ./google_code_execution_files_<YYYYMMDD_HHMMSS>_<random>/ when the first response of the run holds an artifact (0.5.3) and every later iteration writes into the same folder, so a run's artifacts land together and a run without artifacts creates no folder. A later iteration's generated_code_1.py no longer overwrites an earlier one's: it is written as generated_code_1 (2).py (see Output files).

The extractor recognises four part types: executable_code (saved as .py), code_execution_result (.txt), images via as_image() (.png), and — new — any other inline_data blob. Non-image blobs such as CSV, XLSX, PDF, and JSON are now saved under the blob's display_name when the server supplies one (it usually carries the real filename), or a MIME-derived name like generated_file_1.csv otherwise. These blobs were previously discarded silently. Image blobs are still handled by the as_image() branch so nothing is written twice.

Output files: folders and names¶

The same rules apply on OpenAI, Google and Anthropic since 0.5.3:

The default folder is fresh per run. With no code_execution_output_dir, a run that produces files writes them to ./<provider>_code_execution_files_<YYYYMMDD_HHMMSS>_<random>/ in the working directory. The folder is created once the reply cites a file (OpenAI, Anthropic) or a response holds an artifact (Google), before the downloads run, so a run whose downloads all fail leaves an empty folder and a run with no artifact creates none. Before 0.5.3 the folder was stamped to the second, so two runs that started in the same second shared it, and a cited name both produced (output.csv) was silently overwritten with both records reporting success. The random suffix makes sharing impossible; the timestamp keeps the folders sortable, and a glob on <provider>_code_execution_files_* still matches. A folder that cannot be created (a read-only working directory) fails the download records, not the run, on OpenAI; Google's single-request path and Anthropic raise as they always did, and Google's tool-combination loop logs a WARNING and carries on. An explicit code_execution_output_dir is shared across runs by design.
A run never overwrites one of its own files. A name the run already wrote is written as name (2).ext, then name (3).ext, … (the file-manager convention), and the record in result.downloaded_files carries the name actually written in filename. That covers two cited names that clean to one (a:b and a?b both become a_b), a model that saves the same name twice, and Google's generated names across the iterations of a tool-combination run. Only the run's own writes count: a file an earlier run left in a reused explicit folder is overwritten as before, so a client that reruns a job into its own folder still finds report.csv where it looks. A write that fails gives its name back, so the next file of that name is not numbered past one that does not exist; the failed record keeps the name it attempted, with success false and no path, so key successful records by path and failed ones by file_id (OpenAI and Claude; a failed Google record carries neither), never by filename. One INFO line names the rename; the sanitiser's WARNING is reserved for a name that was unsafe.
OpenAI citations are deduplicated. A file the reply cites twice is downloaded once and listed once. result.files_created lists the paths written; the provider's files_created count is one per distinct cited file, successful or not, and files_downloaded the files written.
Anthropic downloads the files its container produced the same way: to code_execution_output_dir, or to a fresh ./anthropic_code_execution_files_<timestamp>_<random>/ — on the single-request path. Claude's SDK streams a download chunk by chunk into the file, so a download that fails midway is removed when the file was new to the folder and the record names what was attempted; an earlier run's file under that name is overwritten in place as before (a failure midway leaves it truncated, as it did before 0.5.3). The tool-calling path (local tools present) downloads the same way since 0.5.3; until then it downloaded only when code_execution_output_dir or skills was set, so a run with code_execution alone and the default folder got no files there.

Anthropic `web_fetch` and programmatic tool calling¶

web_fetch retrieves the contents of a URL. Configure it like any other built-in tool, optionally constraining which hosts it may reach:

agent = Agent(
    model="claude-sonnet-4-5",
    built_in_tools={
        "web_fetch": BuiltInToolConfig(
            enabled=True,
            config={
                "allowed_domains": ["docs.python.org"],
                "max_content_tokens": 20000,
                "citations": {"enabled": True},
            },
        ),
    },
)

`allowed_callers` and older Claude models¶

Anthropic's web_fetch defaults to allowing both direct model invocation and invocation from inside code execution (programmatic tool calling). Models that do not support programmatic tool calling reject that default with a 400, so the toolkit sets allowed_callers=["direct"] automatically for them and logs the substitution at INFO.

Which models need it is registry data, not a hardcoded list — the flag is descriptions.<model>.supports_programmatic_tool_calling in model_information.json. It is currently false for claude-haiku-4-5 only; every other Claude model in the registry supports it, is flagged true and keeps Anthropic's default.

You can always set the field yourself, and an explicit value is never overridden:

BuiltInToolConfig(
    enabled=True,
    config={"allowed_callers": ["direct"]},
)

Reusing Anthropic code execution containers¶

Anthropic code execution runs inside a container whose filesystem and REPL state persist for up to 30 days. By reusing the container id across runs, files uploaded via tool_files in one run remain available in subsequent runs without re-uploading, and variables defined earlier remain in scope. Container reuse is Anthropic-only; other providers silently ignore the settings below.

Where container_id comes from on streaming runs

Code execution is GA and needs no beta header, but the plain /v1/messages streaming response omits the top-level container object — only the beta surface emits it, on message_stop. So whenever code execution is enabled the toolkit opts the request into the code-execution-2025-08-25 beta, which routes the run onto that surface and makes result.container_id populated for streams as well as non-streaming runs. Without it, container_id came back None from a streamed code-execution run and there was nothing to reuse.

Static reuse via `BuiltInToolConfig.config`¶

Pass the prior run's container_id when constructing the Agent:

agent = Agent(
    model="claude-sonnet-4-5",
    instructions="Continue working in the same sandbox.",
    built_in_tools={
        "code_execution": BuiltInToolConfig(
            enabled=True,
            config={"container_id": "container_abc123"},
        ),
    },
)
result = await agent.run("List files in the working directory.")

Per-run override via `anthropic_container_id`¶

Reuse a container for a specific run without rebuilding the Agent:

first = await agent.run(
    "Save a CSV file called data.csv with three rows.",
    tool_files=[...],
)
print(first.container_id)  # capture for reuse

second = await agent.run(
    "Open data.csv and compute the column sums.",
    anthropic_container_id=first.container_id,
)

Auto-enables the code_execution tool when an anthropic_container_id is supplied but the Agent didn't declare the tool — user intent is explicit.

Session auto-reuse¶

Opt in at session creation to have the toolkit capture the container id from the first run and re-inject it into every subsequent run and stream in the session:

async with agent.session(reuse_anthropic_container=True) as s:
    await s.run("Upload and inspect sales.xlsx.", tool_files=["sales.xlsx"])
    await s.run("Now pivot it by region and export pivot.csv.")
    await s.run("Download pivot.csv.")

Non-Anthropic agents log a warning and behave as if reuse_anthropic_container=False.

Upload preservation under container reuse¶

Whenever a container-reuse signal is present — a runtime anthropic_container_id, a BuiltInToolConfig(code_execution, config={"container_id": ...}) binding, or an AgentSession(reuse_anthropic_container=True) — the toolkit automatically tells the Anthropic provider to preserve (i.e. not delete) uploaded tool-file ids and uploaded custom-skill ids at the end of the run. The same ids stay attached to the container on Anthropic's side, so downstream runs that resume the container continue to see the uploaded files and skills without re-uploading.

Without a reuse signal, the toolkit retains its existing behaviour of deleting tool-file and custom-skill ids once the call completes (keeps the Files API workspace tidy for one-shot usage).

Note on filename preservation: when each entry in tool_files carries a filename (directly via dict input, derived from a local file path, or supplied as a filename kwarg on raw bytes), the toolkit forwards it to the provider's Files API so the original name survives into the code-execution container. Anthropic and OpenAI receive a (filename, bytes, content_type) multipart tuple; Google uses the value as the Files API display_name. Without a filename, providers fall back to a generic name (upload for Anthropic, similar defaults for OpenAI, and a sequential tool_file_{i} placeholder for Google), which would cause distinct uploads to collide on the same in-container path within a single run.

Note on same-filename uploads: if you upload a file with the same name to a reused container (e.g. a fresh animals.csv on a follow-up turn), Anthropic places the new bytes at the same path inside the container filesystem, overwriting the prior copy. There is no client-side collision — each upload gets its own Files API id, but the in-container path gets the latest content.

What to validate¶

built-in tool path runs without provider mismatch
output reflects tool-assisted result
schema output is validated when output_schema is provided

Test sources¶

tests/e2e/test_agent_e2e.py (test_built_in_tools_with_thinking, test_built_in_tools_with_structured_output, test_quad_combination_thinking_builtin_custom_tools_schema)
tests/integration/test_combination_openai.py (web search/code execution combinations)
tests/integration/test_anthropic_function_calling.py (built-in tool combinations)