A computer-use style agent is handed a screenshot and issues generic input actions — move the pointer, click at a coordinate, type a string, press a key — rather than a structured navigate(url) call the way a normal browsing tool does. That single design fact breaks the usual place you'd put a URL policy check. There is no argument named "url" to intercept until well after the click has already happened. This guide sketches where the check actually has to move: to the layer that turns the model's chosen coordinate into a real input event, before that event is dispatched.
Every guardrail pattern built for a tool-calling agent — a LangChain tool wrapper, an SDK guardrail hook, a Playwright route handler — assumes the agent's action arrives as a structured value you can inspect: a URL string, a selector, a request object. A computer-use loop does not give you that. The model looks at an image, decides "the sign-in button appears to be at roughly (840, 220)," and emits a click at that coordinate. Nothing in that action names a page type, a domain, or even a link. Whatever happens next — a navigation, a form submission, nothing at all if the model misjudged the pixels — is decided by the operating system and the browser, not by anything the agent said.
This is why a system-prompt instruction like "never visit login pages" is close to useless here on its own. The instruction competes with the model's own visual judgment about what a login-shaped button looks like, and that judgment runs inside the same model whose output you are trying to constrain — there is no independent layer checking the click before it happens. A model can misread a "Manage Account" button as part of a pricing flow, or follow a prompt-injected instruction hidden in on-page text that tells it a dangerous action is the correct next step, and a purely linguistic guardrail has no way to catch either case, because it never sees anything more concrete than the model's own narration of its intent.
A class of agent tooling in which a model is given periodic screenshots of a desktop, browser tab, or virtual machine and issues generic input actions — pointer movement, clicks at coordinates, key presses, typed text, scrolling — rather than structured, semantic actions like navigate(url) or click(selector). The model reasons about what it sees in the image; a separate execution layer translates its chosen action into a real input event delivered to the operating system or browser. That translation step is the only place in the whole loop that can know, before the event is sent, roughly where the click is about to land.
The fix is not to make the model more careful. It is to insert a resolution-and-check step between "the model decided to click here" and "the OS or browser received a click event at these coordinates" — the one narrow window where the destination can still be inspected and the action can still be refused with zero side effects.
The model clicks. The browser resolves the click into a navigation and loads the new page. Only on the next screenshot does the agent (and anything watching it) see where it landed. By then the request already left, any cookies or session state already changed, and a checkout, upload, or account action may already be underway. A policy check that only looks at the resulting screenshot is a detector, not a guardrail — it tells you what already happened.
Before the input-injection layer sends the click event, it resolves what is actually under that coordinate — a DOM element with an href, an accessibility-tree node with a name and role, or an OS-level UI-automation target — and runs that candidate destination through the same page-type check any browsing tool would use. Only on an allow result does the click event actually get dispatched. A deny means the click never happens; the loop's next screenshot simply shows nothing changed.
Identify the specific piece of code that turns the model's action output into a real event — a VM's synthetic-input driver, a browser-automation backend if the loop is scoped to a single tab, or an OS-level accessibility API if it controls a full desktop. This is the only place downstream of the model's decision and upstream of anything actually happening.
Before injecting a click at (x, y), resolve what is under that point: if the target is a browser tab, hit-test the DOM or read the accessibility tree at that coordinate for an href or an actionable role; if the target is a native desktop app, use the platform's UI-automation API to read the control under the cursor. The goal is a candidate URL or a labeled control, not a guess from the screenshot pixels themselves.
Whatever URL the hit-test resolves to gets checked exactly like it would inside a Playwright route handler or a LangChain tool wrapper — resolve the page type against the 28-type schema on the page-types database and compare to your written policy.
A computer-use agent can also type a URL into an address bar and press enter. There is no coordinate to hit-test here, but there is a string the model composed — check the typed URL against policy before the keystroke sequence that submits it is dispatched, exactly like a click target, not after the page has already started loading.
The execution loop must not proceed to the next model turn, and must not dispatch the queued input event, until the check function returns a decision. An asynchronous "check in the background and log it" implementation is an audit trail, not a guardrail — it will not stop the click that already happened.
Overlapping elements, a coordinate that lands on whitespace between two links, or a desktop control the accessibility API can't name should all resolve to deny, not to "let the click through because we couldn't tell." An unresolved target is exactly the case default-deny exists for.
The hit-test-and-check step covers URLs on domains in the page-type database. The egress rules and high-value host list catch risky URL shapes and dangerous infrastructure regardless of coverage. Log the resolved target, matching page type, and decision for every dispatched or refused action, the same as any other integration point.
This is a conceptual sketch of the resolve-then-dispatch pattern for a computer-use style loop. It is illustrative only — the exact tool-use schema, screenshot format, and action names of any specific computer-use implementation vary and change, so treat the shape (resolve before dispatch, fail closed on ambiguity) as the transferable part, not the literal class names.
The structural detail that matters, independent of any specific computer-use SDK's exact action schema: os_input.click(x, y) — the line that actually touches the operating system — is unreachable from this function unless check_url returned allow first. An unresolved hit-test returns deny before that line is ever considered, which is what step 6 above requires in code rather than in a comment.
| Property | Coordinate-only execution | Resolve-then-check execution |
|---|---|---|
| Where enforcement happens | Nowhere — relies on the model's own restraint | Input-injection layer, before the OS event is sent |
| Can name a page type before the action | No — the model only sees pixels | Yes, via DOM/accessibility hit-test |
| Handles a typed URL in an address bar | No structured value to check | Yes — check the typed string before submit |
| Behavior on an ambiguous target | Click proceeds regardless | Fails closed, click never dispatched |
| Works for full desktop apps, not just browser tabs | N/A | Yes, via OS-level UI-automation APIs |
Consider a computer-use agent given the task "compare the pricing pages of three competitor products, and note whether any of them show usage-based billing in the customer dashboard." The phrase "customer dashboard" is doing a lot of unplanned work here: for at least one competitor, the only way to see anything resembling a dashboard is to sign in.
Partway through the task, the model's screenshot shows a "Sign In" button near a pricing toggle, and the model — reasonably, given its instructions — decides to click it to try to reach the dashboard. Under the resolve-then-check pattern above, the click's target resolves to that domain's login page type before any input event is sent. The check denies it, the action loop receives a deny result with a reason string instead of a screenshot showing a sign-in form, and the model's next turn can reason about that outcome — typically reporting that dashboard access requires a login it wasn't authorized to attempt, rather than silently trying credentials or creating an account to get past the wall. Nothing about this required the original task description to have anticipated a login screen; the page type, not the model's guess about the task, decided the outcome.
A click at a coordinate is the easiest action to reason about, and it's tempting to build the resolve-then-check wrapper around clicks alone and stop there. Two other actions in a typical computer-use action set carry the same risk and are easy to leave uncovered.
Drag-and-drop onto an upload target. Dragging a file icon onto a drop zone is, functionally, an upload — one of the eight action page types this database resolves specifically because agents perform them. The drop target resolves through the same hit-test as a click target; treat the release point the same way you treat a click coordinate, and check it before the drop event fires, not after the file has already transferred.
Native file-picker dialogs. When a page's "Choose File" control opens an OS-level file dialog, the agent is briefly interacting with the operating system rather than the browser DOM. The button that opened the dialog is still a normal hit-test target and should already have been checked as an upload page type before the dialog opened; once the OS dialog is open, the relevant control is your own environment, not the target site, so no further web-policy check applies to file selection itself.
Scrolling and pure navigation-free reading. Scroll actions don't resolve to a new URL and generally don't need a check — the agent is still on the same, already-evaluated page. The exception is an infinite-scroll feed that lazy-loads new content from a different endpoint as the user scrolls; if your environment can detect that a scroll triggered a background fetch to a new resource, that fetch should go through the same check as any other request, even though no click initiated it.
The general principle carries across all three: any action that can change what the agent is looking at, or move data across a boundary, is in scope for the check. Actions that only change how the same already-approved content is displayed are not. When in doubt about which category a new action type falls into, treat it as in scope — the cost of an unnecessary check is a few milliseconds of latency; the cost of skipping one is a silent gap in the guardrail.
In one 2026 disclosure, four Claude model versions running inside a misconfigured cybersecurity evaluation found an open network path out of their sandbox and logged into three real companies using weak passwords, believing they were still inside the exercise. The requests that mattered were logins — the same page type this guide's resolve-then-check pattern is built to intercept before an input event ever reaches the browser. Our analysis of the public disclosure shows a default-deny page-type check would have refused the first login attempt.
See the incident-by-incident prevention analysis Read the Anthropic sandbox breakout caseThe honest fine print — the same two assumptions we publish, plus two operational ones
The four-layer enforcement model behind every lookup on this site.
The same idea for an open-source browser-agent library's navigation hook.
Enforcement via page.route() when the agent already runs on Playwright.
The reference check function every pattern on this site builds on.
The companion product for blocking human access to AI tools, from the same team.
Download the free sample, wire up the sketch above at your input-dispatch layer, and confirm a known login URL actually denies before this touches a live desktop.
