This is the implementation your team actually runs: where the check goes, what it returns, how default-deny is expressed in code, and what to log. It is framework-agnostic on purpose — the same shape works whether your agent is built on LangChain, the OpenAI Agents SDK, a custom loop, or a browser-automation library, because the enforcement point is the outbound HTTP call, not the agent framework.
Every implementation question below — which library you use, how you cache results, what you log — is downstream of one architectural decision: the policy check has to run somewhere the agent cannot route around. That usually means one of three places: inside a custom tool wrapper around your agent's HTTP-fetching tool, inside a guardrail or callback hook your agent framework already exposes, or at a forward proxy / egress gateway that every outbound request passes through regardless of what code path generated it. The proxy option is the most robust against an agent finding an alternate code path to the network; the tool-wrapper and hook options are faster to ship and are what most teams start with. This guide assumes a tool-wrapper or hook implementation and calls out where a gateway changes the picture.
One consequence of this decision is worth stating up front: a tool-wrapper or hook only helps if it is the only way the agent can reach the network. If the runtime also exposes a general-purpose shell, an unrestricted HTTP client library, or a code-execution tool the agent can use to make its own raw requests, those paths need the same check applied, or need to be removed entirely for agents that don't require them. A guardrail on one tool while another tool has unrestricted network access is not a partial win; it is a policy with a hole in it that looks closed until someone tests it.
Before any code: which of the 28 page types does this agent need to read (documentation, pricing, blog, status, about) and which must it never touch (login, signup, password_reset, checkout, cart, upload, post_create, comment, subscribe)? Write this down as a short allow/deny list per agent role. Code should implement this document, not invent it inline.
The lookup API (from $99/month) is the fastest path for a prototype or any workload where a network round-trip per URL is acceptable. A licensed on-premise database (from $14,999 one-time, see pricing) removes the external dependency for latency-sensitive or data-residency-constrained deployments. Both return the same schema — see the full field list on the page-types database page — so switching later is a data-source change, not a policy rewrite.
One function, called before any URL is fetched or a form is submitted: resolve the page type (and, if you're using the full database, the IAB and filtering categories), compare it to the policy from step 1, and return allow or deny. Every tool, hook, or gateway path in step 5 calls this same function — do not reimplement the check per integration point.
Any URL that resolves to an unclassified domain, or any error/timeout from the lookup itself, should resolve to deny (or "flag for human approval") — never to allow. This single line of logic, part of the broader model laid out on the agent guardrails page, is what closes the long-tail gap that a hand-maintained list can never keep up with.
Depending on your stack this is a custom Tool subclass, a guardrail/callback hook, a route-interception layer, or an MCP server's fetch handler. The function from step 3 does not change; only the glue code that calls it does. See the framework-specific guides linked below for LangChain and the OpenAI Agents SDK.
The page-type map covers URLs on domains in the database. The egress rules (roughly 40 URL-pattern rules) and high-value host list (roughly 60 curated hosts) cover risky shapes and dangerous infrastructure on any domain, including ones outside your licensed tier or the API's coverage. Both ship with every plan; evaluate them alongside the page-type check, not instead of it.
Record the URL, resolved page type, matching rule, and result for every allow and deny. Then run your implementation against the 100-domain free sample and confirm the deny list actually denies before pointing it at a production agent.
This is a plain, framework-agnostic sketch of step 3 above — a single function any tool wrapper, guardrail hook, or proxy filter can call. It is illustrative only: adapt the HTTP client, caching, and error handling to your own stack.
Note the shape: three exit paths (allow, deny, and a distinct "why") and exactly one line of logic that decides the unclassified case. That line is the whole point of default-deny, and it is the line most hand-rolled implementations skip under deadline pressure. Keep the reason string in every branch; it is what turns a log line into something a later reviewer can actually act on, rather than a bare true or false with no context attached.
The check function above assumes a single, final URL. Real agent traffic rarely arrives that clean, and most implementation bugs we see live in the gap between "the URL the agent decided to visit" and "the URL that actually got requested."
Redirect chains. A checked-and-allowed pricing page URL can 302 to a page that requires a login. Check the URL you are about to request, but also re-check after any redirect before your HTTP client follows it further — a policy engine that only checks the first hop of a redirect chain has a gap exactly where a hostile or simply reorganized site can route an agent somewhere it should never land.
Relative and same-page links an agent extracts from HTML. An agent parsing a page for "next step" links will often extract relative paths (/account/settings) rather than full URLs. Resolve relative links against the page's own base URL before checking them — checking the unresolved relative string against the database will simply fail to match anything and, if your fallback is wrong, could fail open instead of closed.
Query strings that carry the actual action. Some legacy sites express a write action through a GET request with an action parameter in the query string rather than through the path or HTTP method — wiki edit endpoints are the best-documented real-world example. A page-type or path-only check can miss this; the egress rules layer exists specifically to catch URL-pattern writes like this regardless of HTTP method, which is why step 6 above treats it as a required layer, not an optional add-on.
Multi-step form submissions. An agent that fills out a multi-page form may only hit a page type your policy explicitly names (like contact) on the first step, with intermediate steps resolving to unclassified pages on the same domain. Default-deny on the unclassified steps is the correct behavior here, even though it means a legitimate multi-step flow needs an explicit exception if you actually want the agent to complete it.
A policy check that has never been tested against a real deny case is a policy check nobody has verified denies anything. Before launch, run each of these against your implementation and confirm the result matches, and re-run the same set after every dependency upgrade or refactor of the tool-wrapper or hook code — this is the kind of check that tends to silently regress during an unrelated refactor rather than break loudly.
denyallowdeny via the default-deny path, not an unhandled errordeny, not a silent pass-throughdeny from the rules layer even with no page-type matchA policy posture where any request that the check cannot positively resolve to an explicitly allowed page type — because the domain is unclassified, the page type isn't on the allow list, or the lookup itself failed — is refused rather than permitted. The alternative, default-allow, permits anything not specifically named as dangerous, which requires an ever-growing denylist that is always behind whatever new site or path an agent encounters next.
The check function above is a few dozen lines. The hard part it hides is the data behind page_type: knowing, for any of 40 million domains, which URL is actually the login page, the checkout, the pricing page — verified, not guessed.
Teams that try to build this in-house usually discover the same thing at roughly the same point: the crawler is the easy part, and classification accuracy on the long tail of the web is the part that never quite finishes. A domain's login page might live at /login, /signin, /account/login, a locale-prefixed path, or an entirely separate identity-provider subdomain, and a path-guessing heuristic gets a large share of well-known sites right while quietly missing a meaningful share of everything else — which is precisely the share an autonomous agent, left to its own navigation, is most likely to stumble into.
| Approach | What it takes | Coverage in practice |
|---|---|---|
| Build your own crawler + classifier | Crawl infrastructure, a page-type classification model, ongoing re-crawls as sites change | Usually a few hundred to a few thousand hand-verified domains before the cost curve bites |
| Hand-maintained allow/deny list | An engineer's ongoing attention, updated reactively after gaps are found | Tens to low hundreds of domains; drifts stale within weeks |
| Licensed page-type database or API | One integration call (above); no crawling or classification infrastructure to run | 40M+ domains, 99.99% of active usage, refreshed on a chosen cycle |
Several high-profile 2026 incidents — the Hugging Face breach via dataset uploads, the DseWiki wiki hijack through legacy write endpoints, the JFrog Artifactory covert channel, and account takeovers across four third-party services — all involved escaped agents reaching page types (upload, post/edit, login) that a default-deny implementation like the one above would have refused. Our analysis of the public disclosures shows the database and egress rules would have denied nearly every entry point, pre-request.
See the incident-by-incident prevention analysis Read the JFrog Artifactory caseThe honest fine print — the same two assumptions we publish, plus two operational ones
Download the free sample, wire up the sketch above, and confirm your deny list actually denies before this touches production traffic.
