Skip to content

Recipe: Approving Browser and Desktop Actions with Hooks

Goal

Let an agent drive a website or a desktop through the doc-processing-std-tools browser and computer servers while your application decides which actions run: pause for a human before every browser action (a click, typing, a key, a form input, an upload, a script), refuse navigation off an allow-list with a reason the model reads, pause before every desktop action on the user's own screen, and keep an audit line naming the element each action touched.

Prerequisites

  • doc-processing-std-tools 0.6.0 with its browser server (servers/browser) or computer server (servers/computer) — attached in-process (from doc_processing_std_tools.browser import browser_server, then Agent(tools=[browser_server])), or through the composed launcher with DPTTOOLS_ENABLE_BROWSER=true / DPTTOOLS_ENABLE_COMPUTER=true in the server's env (an mcp_command server does not inherit this process's variables; see Tools).
  • The toolkit's hooks channel (Hooks and steering).

Why approval belongs to the app

std-tools has a server-side confirmation gate for consequential actions — DPTTOOLS_BROWSER_CONFIRM and DPTTOOLS_COMPUTER_CONFIRM — and it is off by default: the servers expect the client application to approve actions through its own hooks (Security). The toolkit gives you one place to do that: before_tool_call, the hook event that fires immediately before a tool is dispatched.

  • A pause returned there blocks the run before the tool runs, until the host calls agent.run_handle.resume() (the action then runs) or agent.run_handle.abort(reason) (it never runs and the run ends with AgentAborted).
  • A skip_tool declines the call; its message is what the model reads in the tool's slot, so the model can take another route instead of failing.
  • A hook sees the call as the model made it — browser_left_click(ref="ref_12"), computer_left_click(x=412, y=230) — not "button 'Place order'". The browser tools name the element they acted on in the first line of their result ("Clicked button 'Place order'"), which is what to log; the desktop tools see pixels and cannot name anything.

The two hook classes below are the ones tests/unit/test_hooks_recipe_examples.py drives (their source is tests/fixtures/hooks_recipe_examples.py); copy them into your app and adjust the sets.

The browser hook

import logging
import re
from typing import Callable, Iterable, Optional
from urllib.parse import urlparse

from doc_processing_toolkit.agent.cost import AgentStreamEvent
from doc_processing_toolkit.agent.hooks import HookDirective

#: Browser tools that only look, read, scroll, hover or manage the agent's own tabs. Every other
#: ``browser_*`` tool acts on the page and pauses for a human's yes — a tool this set does not name
#: (one std-tools adds later, or one you forgot) is gated, not let through: the hook fails closed.
BROWSER_LOOK_ONLY_TOOLS = frozenset(
    {
        "browser_screenshot",
        "browser_zoom",
        "browser_hover",
        "browser_mouse_move",
        "browser_scroll",
        "browser_scroll_to",
        "browser_wait",
        "browser_read_page",
        "browser_find",
        "browser_get_page_text",
        "browser_list_tabs",
        "browser_switch_tab",
        "browser_close_tab",
        "browser_read_console",
        "browser_read_network",
    }
)
#: Every tool of the browser server that opens a URL the model names; all take ``url``.
NAVIGATION_TOOLS = frozenset({"browser_navigate", "browser_new_tab", "fetch_webpage_rendered"})
#: ``browser_navigate`` also takes these instead of a URL; they stay on pages already approved.
NAVIGATION_COMMANDS = frozenset({"back", "forward", "reload"})


def domain_of(url: object) -> str:
    """``"https://Shop.example.com/cart"`` → ``"shop.example.com"``; ``""`` when there is no host.

    A scheme-less URL is read as std-tools reads it, with ``https://`` in front.
    """
    text = str(url or "").strip()
    if "://" not in text:
        text = "https://" + text
    return (urlparse(text).hostname or "").lower()


def on_allow_list(domain: str, allowed: Iterable[str]) -> bool:
    """Whether *domain* is one of *allowed* or a subdomain of one."""
    return any(domain == a or domain.endswith("." + a) for a in (d.lower() for d in allowed))


def first_result_line(meta: dict) -> str:
    """The first line of an MCP tool result's text, from a ``tool_call_result`` event's metadata.

    With ``Agent(tool_result_preview_chars=0)`` the event carries ``result_data`` — the text
    itself for a one-item result, the list of content items otherwise — which is the reliable
    source. Without it, ``result`` is the first 500 characters of the result's ``str()``: for an
    MCP tool the repr of ``{'content': [{'type': 'text', 'text': ...}]}``, whose text is read out
    with whichever quote Python chose for it (double quotes when the text holds an apostrophe).
    """
    text = _first_text(meta.get("result_data"))
    if text is None:
        text = _text_from_repr(str(meta.get("result") or ""))
    return (text or "").split("\n", 1)[0] or "(no result text)"


def _first_text(data: object) -> Optional[str]:
    """The first text of a ``result_data`` value: a string, a text content item, or the first of a list.

    A JSON object a tool returned (``tool_result_data`` decodes one) is not read by its own ``text`` field.
    """
    if isinstance(data, str):
        return data
    if isinstance(data, list):
        for item in data:
            found = _first_text(item)
            if found is not None:
                return found
        return None
    if isinstance(data, dict) and data.get("type") == "text" and isinstance(data.get("text"), str):
        return data["text"]
    return None


_REPR_TEXT = re.compile(r"'text': (['\"])((?:\\.|(?!\1).)*)\1?", re.DOTALL)
_REPR_ESCAPES = {"n": "\n", "t": "\t", "r": "\r", "\\": "\\", "'": "'", '"': '"'}


def _text_from_repr(raw: str) -> str:
    """The first ``'text'`` value of a result repr, unescaped; the whole string when there is none."""
    match = _REPR_TEXT.search(raw)
    if not match:
        return raw
    return re.sub(r"\\(.)", lambda m: _REPR_ESCAPES.get(m.group(1), m.group(0)), match.group(2))


def pending_call(meta: dict) -> dict:
    """What a paused run is waiting on: ``{"tool", "tool_call_id", "arguments"}`` from the event."""
    return {
        "tool": meta.get("tool_name") or "",
        "tool_call_id": meta.get("tool_call_id"),
        "arguments": dict(meta.get("arguments") or {}),
    }


class ApproveBrowserActions:
    """Pause before an action; refuse navigation off the allow-list; log what acted.

    ``before_tool_call`` is the one point a tool can still be stopped: a ``pause`` here blocks the
    run *before* the tool is dispatched until the host calls ``agent.run_handle.resume()`` (the
    action then runs) or ``.abort(reason)`` (it never runs). ``RunHandle`` says only *that* the
    run is paused, so the call being asked about is kept in ``pending`` for the host to show the
    user. A ``skip_tool`` declines the call and its ``message`` is what the model reads in the
    tool's slot, so it can take another route. The gate fails closed: only the tools named in
    ``BROWSER_LOOK_ONLY_TOOLS`` run without a yes, so a tool std-tools adds later pauses too.
    std-tools' own gate (``DPTTOOLS_BROWSER_CONFIRM``) is off by default: this is where approval
    lives. The hook sees every navigation the model *asks for*; what it cannot see — a link the
    model clicks, a redirect — is for the server's ``DPTTOOLS_BROWSER_ALLOWED_DOMAINS``.
    """

    def __init__(self, allowed_domains: Iterable[str], log: Optional[Callable[[str], None]] = None) -> None:
        self.allowed_domains = tuple(allowed_domains)
        self.log = log or logging.getLogger("app.browser_audit").info
        self.audit: list[str] = []
        #: The call the run is paused on, until its result arrives; a host that aborts clears it itself.
        self.pending: Optional[dict] = None

    async def on_event(self, event: AgentStreamEvent) -> Optional[HookDirective]:
        meta = event.metadata or {}
        tool = meta.get("tool_name") or ""
        if not (tool.startswith("browser_") or tool in NAVIGATION_TOOLS):
            return None

        if event.event_type == "before_tool_call":
            arguments = meta.get("arguments") or {}
            if tool in NAVIGATION_TOOLS:
                target = str(arguments.get("url") or "").strip()
                blank_tab = tool == "browser_new_tab" and not target  # its url is optional: a blank tab
                command = tool == "browser_navigate" and target.lower() in NAVIGATION_COMMANDS
                domain = domain_of(target)
                if not (blank_tab or command or on_allow_list(domain, self.allowed_domains)):
                    if tool == "fetch_webpage_rendered":
                        self._record(f"{tool}: refused {target or '(no url)'}")
                    return HookDirective(
                        action="skip_tool",
                        message=f"Navigation to {domain or 'that address'} is not allowed in this app; "
                        f"stay on {', '.join(self.allowed_domains)}.",
                    )
                if tool == "fetch_webpage_rendered":
                    # its result is the page itself (JSON whose text is the content), not a line about
                    # an action: audit the request instead
                    self._record(f"{tool}: {target}")
            elif tool not in BROWSER_LOOK_ONLY_TOOLS:
                self.pending = pending_call(meta)
                return HookDirective(action="pause", reason=f"{tool} needs the user's approval")

        elif event.event_type == "tool_call_result" and tool != "fetch_webpage_rendered":
            if self.pending and self.pending["tool_call_id"] == meta.get("tool_call_id"):
                self.pending = None
            # std-tools' pointer and form tools name the element they acted on in the first line of
            # their text ("Clicked button 'Place order'"); keep that line for the audit log. A call
            # this hook declined comes back the same way, carrying the decline text, so refusals are
            # audited too.
            self._record(f"{tool}: {first_result_line(meta)}")
        return None

    def _record(self, line: str) -> None:
        self.audit.append(line)
        self.log(line)

What it does, event by event:

Call Directive What happens
browser_navigate, browser_new_tab or fetch_webpage_rendered to shop.example.com; browser_navigate to back / forward / reload; browser_new_tab with no URL (a blank tab) none proceeds
the same three to a domain off the allow-list skip_tool not run; the model reads "Navigation to pay.other.net is not allowed in this app; stay on example.com."
every other browser_* tool — the clicks, browser_type, browser_key, browser_form_input, browser_file_upload, browser_javascript_exec, browser_run_script, and any tool std-tools adds later pause the run blocks before the action; pending names the call; the host resumes or aborts
the look-only set: browser_read_page, browser_get_page_text, browser_screenshot, browser_find, browser_scroll, browser_hover, the tab tools, the console and network reads none reading, looking and moving about are never gated
any browser_* result none the first line of the result's text (the element name) goes to the audit log — Clicked link 'Learn more' [ref_1]. in a live run. A navigation the hook declined comes back as a result too, carrying the decline text, so the refusal is audited; an aborted pause leaves no result — the run ends. fetch_webpage_rendered is audited by the URL it asked for (allowed or refused) instead, since its result is the page itself

Build the agent with tool_result_preview_chars=0 and the event carries result_data — the result's text itself, or its list of content items — which first_result_line reads first. Without it the event's result is the first 500 characters of the result's str(), for an MCP tool the repr of a {'content': [...]} dict, and the text is read out of that with whichever quote Python chose for it (see Tool results in events); a line cut at 500 characters is cut in the audit too. With tool_result_preview_chars=0 every event carries the whole result — a browser_screenshot's base64 image included, and every acting tool's when DPTTOOLS_BROWSER_AUTO_SCREENSHOT is on — so keep the default where events are logged or stored, as that page warns.

The gate fails closed. BROWSER_LOOK_ONLY_TOOLS names what may run without a yes — reading, looking, scrolling, hovering, the agent's own tabs, the console and network reads — and every other browser_* tool pauses: the clicks, browser_type and browser_key (on by default, so a model can type into a field and press Enter), browser_form_input, the opt-in browser_file_upload, browser_javascript_exec and browser_run_script, and any tool std-tools adds later, which a list of tools to gate would have let through. Move a tool into the look-only set only once you have read what it does. The allow-list is applied to every tool that opens a URL the model names: browser_navigate, browser_new_tab and fetch_webpage_rendered (registered on the browser server under both profiles). The hook sees every navigation the model asks for; what it cannot see — a link the model clicks, a redirect — is for the server: set DPTTOOLS_BROWSER_ALLOWED_DOMAINS as well, and the two lists together cover both routes. std-tools lists which tools act in servers/browser.

Wiring it up and answering the pause

import asyncio

from doc_processing_toolkit import Agent
from doc_processing_toolkit.agent.hooks import AgentAborted

approvals = ApproveBrowserActions(allowed_domains=["example.com"])

agent = Agent(
    model="claude-sonnet-5",
    instructions="Fill in the enquiry form on example.com with the details you are given.",
    tool_result_preview_chars=0,   # the events carry result_data: the tool's text, not a repr
    tools=[
        {
            "type": "mcp_command",
            "command": "python",
            "args": ["-m", "doc_processing_std_tools"],
            "name": "std-tools",
            "env": {"DPTTOOLS_ENABLE_BROWSER": "true", "DPTTOOLS_BROWSER_ALLOWED_DOMAINS": "example.com"},
        }
    ],
    hooks=approvals,
)


async def main():
    task = asyncio.create_task(agent.run("Open https://example.com/contact and submit the form for Ada Lovelace."))
    try:
        # Elsewhere in the same process — a UI callback, a queue consumer — once the user has answered.
        # RunHandle says only that the run is paused; the hook's pending says on which call.
        while not task.done():
            await asyncio.sleep(0.2)
            handle, call = agent.run_handle, approvals.pending
            if handle is not None and handle.paused and call is not None:
                question = f"The agent wants to run {call['tool']} with {call['arguments']}. Allow?"
                if await ask_user(question):                           # your UI; it may raise or time out
                    handle.resume()                                    # the action runs
                else:
                    handle.abort(f"the user declined {call['tool']}")  # AgentAborted ends the run
                    approvals.pending = None                           # no result will come to clear it
        result = await task
    except AgentAborted as exc:          # the user declined: the run ended before the action
        print("aborted:", exc.reason)
    finally:
        if not task.done():              # the UI failed while the run was paused: end the run first
            if agent.run_handle is not None:
                agent.run_handle.abort("the host could not ask the user")
            await asyncio.gather(task, return_exceptions=True)
        await agent.close()              # disconnects the std-tools server (and its Chromium) either way
    print(approvals.audit)               # ["browser_left_click: Clicked button 'Send'", ...]


asyncio.run(main())

close() matters for a subprocess server — hence the finally, which runs after an abort too and, when your UI raises while the run is paused, first aborts the run and waits for it (a paused run cannot be closed from under itself). The Agent keeps the connection open for its next run(), and a process that exits with it open tears the session down during event-loop shutdown, which anyio reports as a cancel-scope error. See MCP session lifecycle.

RunHandle says only that the run is paused, not on which call: the hook's pending — {"tool", "tool_call_id", "arguments"} — is what the host shows the user, and it is cleared when that call's result arrives (after an abort no result comes, so the wiring clears it). pause is in-process and holds the run's resources while it waits, so give the confirmation a timeout policy (abort after a few minutes of silence) — see Controlling a live run with RunHandle. To decline one click without ending the run, make the decision inside the hook instead: await your UI's answer there and return skip_tool with the reason, or None to proceed — a hook is awaited, so blocking in it holds the run exactly as pause does.

Running both hooks together

One hooks= value is one hook; to run the browser and desktop hooks together, dispatch from a small composite:

class AllOf:
    def __init__(self, *hooks):
        self.hooks = hooks

    async def on_event(self, event):
        for hook in self.hooks:
            directive = await hook.on_event(event)
            if directive is not None and directive.action != "continue":
                return directive          # the first hook with an opinion decides
        return None

browser_approvals = ApproveBrowserActions(["example.com"])
desktop_approvals = ApproveDesktopActions(host_mode=True)   # keep both names: their pending is what you show
agent = Agent(..., hooks=AllOf(browser_approvals, desktop_approvals))

The desktop hook

The computer server operates a desktop: its own virtual display by default, or the user's real screen in host mode (DPTTOOLS_COMPUTER_BACKEND=host; see Isolated vs attached). It sees pixels, not element names, so std-tools' own gate can do less here and approval in the app matters more.

#: Computer tools that only look or wait; every other ``computer_*`` tool acts on the desktop.
COMPUTER_LOOK_ONLY_TOOLS = frozenset(
    {"computer_screenshot", "computer_zoom", "computer_cursor_position", "computer_wait"}
)
#: Key names std-tools folds together before it judges a chord, so ``Control+W`` is ``ctrl+w``.
_KEY_SYNONYMS = {
    "control": "ctrl",
    "ctl": "ctrl",
    "option": "alt",
    "cmd": "super",
    "command": "super",
    "meta": "super",
    "win": "super",
    "windows": "super",
    "del": "delete",
}
#: Key chords that close, quit or delete — the ones ``DPTTOOLS_COMPUTER_CONFIRM=deny`` refuses
#: server-side — as sets of canonical keys, so ``F4+Alt`` and ``alt+f4`` are one chord.
REFUSED_KEY_CHORDS = frozenset(
    {
        frozenset({"alt", "f4"}),
        frozenset({"ctrl", "w"}),
        frozenset({"ctrl", "q"}),
        frozenset({"super", "w"}),
        frozenset({"super", "q"}),
        frozenset({"delete"}),
        frozenset({"shift", "delete"}),
    }
)


def chord_keys(chord: str) -> frozenset[str]:
    """``"Shift+Del"`` → ``{"shift", "delete"}``: a chord read as std-tools reads it.

    Split on ``+``, lower-cased, synonyms folded, order ignored.
    """
    return frozenset(_KEY_SYNONYMS.get(key.lower(), key.lower()) for key in chord.split("+") if key)


class ApproveDesktopActions:
    """In host mode, pause before every computer tool that acts; always refuse the closing chords.

    The desktop server sees pixels, not element names, so a hook sees ``computer_left_click(x=412,
    y=230)`` and cannot tell what the click will do — approval belongs to the user who can see the
    screen. In ``host`` mode (``DPTTOOLS_COMPUTER_BACKEND=host``, the user's own screen) every acting
    tool pauses, and ``pending`` holds the call for the host to show; in the default ``virtual`` mode
    (the agent's own display) only the closing chords are refused, which mirrors
    ``DPTTOOLS_COMPUTER_CONFIRM=deny`` on the app side.
    """

    def __init__(self, host_mode: bool) -> None:
        self.host_mode = host_mode
        #: The call the run is paused on, until its result arrives; a host that aborts clears it itself.
        self.pending: Optional[dict] = None

    async def on_event(self, event: AgentStreamEvent) -> Optional[HookDirective]:
        meta = event.metadata or {}
        tool = meta.get("tool_name") or ""
        if event.event_type == "tool_call_result":
            if self.pending and self.pending["tool_call_id"] == meta.get("tool_call_id"):
                self.pending = None
            return None
        if event.event_type != "before_tool_call":
            return None
        if not tool.startswith("computer_") or tool in COMPUTER_LOOK_ONLY_TOOLS:
            return None

        if tool in {"computer_key", "computer_hold_key"}:
            chords = str((meta.get("arguments") or {}).get("text") or "").split()
            refused = [chord for chord in chords if chord_keys(chord) in REFUSED_KEY_CHORDS]
            if refused:
                return HookDirective(
                    action="skip_tool",
                    message=f"The key chord {', '.join(refused)} closes, quits or deletes and is not allowed here.",
                )

        if self.host_mode:
            self.pending = pending_call(meta)
            return HookDirective(action="pause", reason=f"{tool} on the user's screen needs their approval")
        return None

chord_keys reads a chord the way std-tools does — split on +, lower-cased, Control / Cmd / Del folded to ctrl / super / delete, order ignored — so F4+Alt and Control+W are refused like alt+f4 and ctrl+w: exactly the chords DPTTOOLS_COMPUTER_CONFIRM=deny refuses. In host mode pending holds the paused call, as the browser hook's does.

To show the user what is about to happen, keep the latest computer_screenshot result and display it with the click's coordinates, or ask the model to state its purpose in each call's optional intent argument (std-tools logs it; your hook sees it in arguments). In host mode get the user's consent before the agent starts: everything on their screen goes to the model provider.

The server-side alternatives

For an unattended pipeline with nobody to answer a pause, let std-tools refuse instead:

  • DPTTOOLS_BROWSER_CONFIRM=deny refuses clicks on elements whose name reads like a purchase, payment, deletion, sending or publishing, account closure or accepting terms, as well as password entry, uploads and scripts (a word-list heuristic on the element name).
  • DPTTOOLS_COMPUTER_CONFIRM=deny refuses the key chords that close, quit or delete; it cannot tell what a click will do.
  • Both gates also take elicit, which asks a human through MCP elicitation. The toolkit does not support elicitation today, so with the toolkit elicit refuses every flagged action — and on the computer server's host backend every call, screenshots included, because the first thing it asks is whether the agent may control the computer at all. When the toolkit supports elicitation, elicit becomes the server-side form of the pauses above.

Both variables are set in the server's environment — the config's env for an mcp_command / mcp_script server — and documented on the std-tools configuration page. The two layers compose: the server refuses what it can recognise, the hook pauses for what only a person can judge.