A domain-level allow or deny list has exactly one bit of information per domain: yes or no. That is enough for a domain that only does one thing. It falls apart the moment a domain hosts a marketing page, a support center, a login form, and a checkout flow all under the same hostname — which is most of the commercially interesting internet. This page works through the amazon.com problem in depth, with several more multi-purpose domains, so the failure mode is concrete rather than theoretical.
Most web-access policy tools, including the ones agent platforms inherit from years of employee-web-filtering practice, were built around the domain as the smallest thing you can allow or deny. That assumption is reasonable for a single-purpose domain. It breaks down for anything larger.
Think about what a domain-level rule actually has to decide. "Allow example.com" or "deny example.com" is a single verdict applied to every URL that hostname will ever serve, today and in the future. For a small brochure site with no login and no forms, that single verdict is fine because there is genuinely only one kind of page there.
The domains an agent actually needs to visit for research, procurement, sales, or support work are almost never that simple. A single hostname commonly serves a public marketing site, a documentation hub, a support center, a billing portal, an account dashboard, and a payment flow — twenty-eight distinct kinds of page, in our own taxonomy, that a single domain-level rule cannot tell apart.
An agent allowed onto "amazon.com" because a procurement task needs product and pricing information is, under a domain rule, also allowed onto the login page, the checkout flow, the seller account console, and the returns and refunds workflow — because the rule only knows the hostname, not the path. A domain denied because a security team is nervous about e-commerce sites is also denied from the help center article that would have answered the agent's question in one read. Both outcomes are wrong, and a domain-level rule cannot avoid producing one or the other on any domain that mixes page types.
This is not a new problem invented by AI agents; the same limitation existed for employee web-filtering and for RPA scripts before it. What is new is the stakes. A human employee who is denied a page notices and asks IT for an exception. An RPA script follows a fixed sequence of selectors someone tested in advance. An autonomous agent decides, at run time and without a human in the loop, which link on the page it just fetched to click next — which means the difference between "allow this domain" and "allow this page type" is now the difference between an agent that reads a pricing page and one that submits a login form, with nobody watching the decision happen in between.
amazon.com is the canonical illustration because almost everyone has an intuitive model of what the site contains: a storefront, obviously, but also an account system, a marketplace seller console, a media platform, and more — all under one hostname. Map even a partial list of its page types by risk and the problem is immediate.
Eight page types, three completely different risk levels, one hostname. A rule that says "allow amazon.com" or "deny amazon.com" cannot land on the correct answer for all eight rows at once, because there is no single correct answer at the domain level — the correct answer only exists at the page-type level. This is exactly what our 28-page-type schema on the page-types database page is built to resolve: a lookup on any of these paths returns the page type, not just the hostname.
Once you look for it, the pattern is everywhere. Here are four more domain archetypes an agent runs into constantly, each one impossible to reduce to a single allow or deny verdict without either over-blocking legitimate research or under-blocking a genuine risk.
| Domain archetype | Page types coexisting on it | Highest-risk page type present | What a domain-level rule gets wrong |
|---|---|---|---|
| A SaaS vendor's marketing + product domain | Pricing, documentation, integrations, blog, careers — and signup, login, subscribe, checkout for the paid tiers | Signup / checkout (creates an account or starts billing) | Allow it and a vendor-research agent can also open a trial account or start a subscription on your behalf. Deny it and the same agent cannot read the pricing page it was sent to check. |
| A wiki or knowledge base with public editing | Read-only articles, a search index, a talk/discussion namespace, and a write-enabled edit form reachable from every page | post_create / comment (an edit or talk-page write) | Wikis look purely informational, so domain-level tools rarely flag them. The edit endpoint is often a simple GET-based form nothing about the domain's reputation would surface. |
| A marketplace or classifieds site | Public listings, seller profiles, a buyer messaging system, and a seller-account dashboard with payout settings | upload / post_create (a new listing) and login (seller account) | An agent scraping listings for market research looks identical, at the domain level, to one about to message a seller or edit a listing — the two need opposite policies. |
| A bank or financial institution's public site | Marketing pages, a branch locator, a careers section, press releases — and online banking login, transfers, and card management behind the same hostname | login and any transfer/payment page type | Security teams often deny financial-services domains outright, which also blocks a compliance-monitoring agent from reading the bank's own published disclosures and legal pages. |
Every one of these archetypes has the same shape as the amazon.com example: a handful of page types that are safe to browse without limit, sitting on the same hostname as a handful of page types where an agent taking an action would matter a great deal. The database behind AI Agent Allowlist resolves this at the URL level across 28 page types and 40 million-plus domains, precisely so a policy engine never has to guess which archetype a given hostname falls into.
Notice, too, that the four archetypes above cover very different industries and very different reasons an agent might visit them — competitive research, market scanning, recruiting, compliance review — which is the point. This is not a narrow problem limited to shopping agents or one vertical. Any agent whose job involves reading part of a domain while never touching another part of that same domain runs into this exact structural gap, and the gap does not shrink as the number of domains an agent needs to cover grows; it multiplies, because each new domain is its own small instance of the amazon.com problem.
The natural next move, once the problem above is visible, is to reach for a domain reputation score or a content-category feed and assume it solves this. It does not, because it answers a different question.
These are not hypothetical edge cases. They are the predictable, structural consequences of treating a domain as one unit of risk when it is not.
Notice that all four failure modes below point in the same direction: a domain-only policy forces a choice between being too permissive or too restrictive, and the choice has to be made once, in advance, for every page the domain will ever serve. Page-type resolution removes the forced choice by letting the answer differ per page, which is the entire reason the distinction in this article matters operationally rather than just conceptually.
A security team, wary of e-commerce or financial domains as a category, denies them outright. A procurement or compliance agent then cannot read a vendor's own published pricing or legal disclosures, and someone has to manually approve an exception for every domain that comes up — which does not scale past a handful of vendors.
The opposite failure is more dangerous and less visible: a large, well-known, "trusted" domain gets allowed wholesale because nobody wants to block a household name. The agent now has standing access to that domain's login form and checkout flow too, with nothing in the policy stopping it.
A vendor adds a self-serve signup flow, a wiki adds a new namespace, a SaaS product ships an in-app upload feature. None of this changes the domain's reputation score, so a domain-level allowlist keeps its old verdict while the actual risk surface underneath it has changed.
A sales-intelligence agent, a price-monitoring agent, and a procurement agent legitimately need different levels of access to the very same domain. A single domain-level verdict per hostname cannot express three different policies for three different agent roles on one site — page-type resolution can, because each role's policy is just a different allow/deny mapping over the same 28 types.
Several high-profile 2026 incidents involved AI agents finding a writable surface on a domain that looked purely informational at the domain level — a legacy wiki with a GET-based edit endpoint being the clearest case. Our analysis shows the database and egress rules would have denied almost all of these entry points before the first request, because the check happens per page type, not per hostname.
Every 2026 agent escape, mapped to the rule that stops it Read the DseWiki case in fullThe honest fine print — the same two assumptions we publish, plus two operational ones
Why an instruction not to visit certain pages is not itself a control.
A generic framework for how task, drift, surface, and action combine.
The four-layer enforcement model behind every lookup.
The 100M+ domain filtering dataset from the same team, for inbound/human traffic.
Download the sample and see what a page-level lookup returns for a domain you already thought you understood.
