AI gateways have earned their place in the stack by inspecting prompts, tool calls, and model output for injection, leakage, and policy violations. Almost none of them answer a simpler question: when an agent is about to fetch a specific URL, is that URL a login page, a checkout, a comment box, or a safe read? AI Agent Allowlist is the page-type lookup that fills that gap — one call your gateway can make on every outbound navigation, across 40 million+ domains.
Gateways already sit in the request path. Adding this check costs one lookup, not a new interception point.
This is written for teams building or operating an AI gateway, guardrail engine, or agent-traffic proxy that customers route their agent fleets through — not for end users of any one agent product. If that describes your architecture, the rest of this page is a technical and commercial case for adding one lookup to your existing request pipeline.
Your customers were exposed to this. Gateways that inspected prompts and tool-call arguments in 2026 still let agents reach wiki-edit endpoints, third-party logins, and cluster-admin surfaces — because none of those inspections looked at what kind of page the URL actually pointed to.
A gateway’s prompt and tool-call inspection answers “is this request trying to manipulate the model?” Page-type policy answers a completely different question: “independent of intent, what is this specific destination, and should any agent be allowed there?” A perfectly well-intentioned agent with a clean prompt history can still be one ambiguous task away from a login form. Catching that requires knowing what the URL is, not what the prompt says.
Prompt injection detection, tool-call argument validation, output filtering for secrets and PII, rate limiting, and model routing. This is real, necessary coverage — and it stops at the boundary of the request’s content, not its destination.
An agent with a syntactically clean tool call — fetch(url) — passes every existing gateway check even when the URL is a login page or a checkout flow, because none of those checks classify the destination itself.
Resolve the URL against the page-type map before the fetch executes. The gateway gets a page_type, an IAB category, and a filtering category back — enough to allow, deny, or flag, in the same policy language it already uses for everything else.
Take a gateway product sold to enterprises running customer-support and sales-research agents through it. The gateway already terminates every outbound HTTP call the agents make, so it is the natural place to add this check — not a new proxy, not a new SDK, just an additional lookup inside middleware that already runs on every request.
A support agent is told to check a vendor’s status page and summarize an outage. Its browsing tool resolves a URL that turns out to be the vendor’s status page — page type status, allow. On the same task, a differently phrased instruction leads the agent toward what looks like a contact form but resolves to the vendor’s account-login page instead — page type login, deny, logged with a reason. Neither request looked suspicious to a prompt-injection filter; both requests were syntactically ordinary tool calls. The only signal that distinguished them was what page type the URL actually resolved to — which is precisely the signal a gateway does not have today without this layer.
The same pattern applies to sales-intelligence agents pulled toward a target company’s leadership and case-study pages (allow) versus its subscribe or account-signup forms (deny), and to procurement-research agents reading a vendor’s pricing and documentation (allow) while being kept off its cart and checkout flow (deny, unless the vendor is on an approved-purchasing list your customer configures). In every case, the distinguishing signal is the page type of the exact URL, not the phrasing of the prompt that led there.
Page-type classification covers the 40M+ domains in the database. Two companion layers, included with every database license, extend coverage to surfaces that can appear on any domain at all — which matters for a gateway seeing traffic to sites nobody pre-classified.
Roughly 40 URL-pattern rules matching risky request shapes — wiki edit endpoints, WebDAV writes, plugin installs, signup-shaped forms — evaluated against the exact URL and method, on any domain, including ones your page-type coverage does not reach. Published in full at the rules file.
Roughly 60 curated dangerous-infrastructure hosts — cloud metadata endpoints, package registries, tunnel and webhook sinks — each with a default result a gateway can apply before any classification lookup even runs. Listed at the host list.
Anything not matched by a page type, an egress rule, or the host list falls to whatever default your gateway configures — deny is the recommended posture for agent traffic, since an unclassified destination is itself a signal worth treating with caution.
Together the four layers are the same framework this site describes on agent guardrails: page types for the 40M+ domains you have classified, egress rules and the host list for everything else, and default-deny as the backstop. A gateway implementing all four has a materially stronger answer to a customer’s security questionnaire than one implementing prompt inspection alone.
Neither layer replaces the other. A gateway that runs both closes a boundary that either one alone leaves open.
| Control | What it inspects | Catches a hostile prompt? | Catches a login/checkout URL? |
|---|---|---|---|
| Prompt injection detection | Content of the incoming prompt and page text | Yes | No |
| Tool-call schema validation | Shape and type of the tool arguments | Partially | No |
| Output / secret filtering | What the model returns to the user | No | No |
| Domain-level allow/deny lists | Whether the domain itself is on a list | No | Only if the whole domain is blocked |
| Page-type URL policy (this database) | What page type the exact URL resolves to | Not its job | Yes, before the request fires |
The useful reframe for gateway product teams: prompt-layer controls protect the model from the page; URL-layer controls protect the world from the agent. Both belong in the same product, and most gateways today ship only the first.
There is a second, quieter reason gateways under-invest in URL-layer controls: it looks, at a glance, like something a domain blocklist should already handle. It does not. Domain-level lists force an all-or-nothing choice per site — block amazon.com entirely, or allow it entirely and hope the agent never finds the checkout. Page-type policy is the missing granularity: allow the domain, deny the specific page type, on the same request path a gateway already owns.
Gateways sit in front of many agents, many teams, and often many different model providers. A page-type check added at the gateway layer becomes a single control that governs all of them at once — rather than a policy each team has to reimplement inside its own agent.
The pattern below is intentionally generic — it is the shape of the check regardless of which gateway product or proxy framework it runs inside.
Note the deny-with-reason shape in the pseudocode above: the response carries not just a verdict but the page type that triggered it. That reason string is what your agent’s own reasoning loop can act on — try a different, allowed page instead of retrying the same denied URL — and what your audit log stores for later review, so a denial is never just a silent drop.
The GET request pattern is deliberate: GET https://www.aiagentallowlist.com/api/check?url=https://stripe.com/login returns "result":"deny","id":"login" in one round trip — no SDK, no session state, easy to bolt onto an existing middleware chain regardless of language. Full request and response shapes are in the API docs.
deny, flag, and allow per page type, defaulting to deny for identity and commerce types/api/check, or a local index if you license the full databaseNone of this requires touching your prompt-injection or tool-validation code paths. It is an additional, independent check that runs in parallel and only needs to agree with your existing controls at the point of final decision: deny wins.
Most gateway integrations take one to two sprints from a standing start: a day or two to wire the API call into the existing interception point, a sprint of shadow-mode logging against real customer traffic to see what the deny set would actually catch, and a final sprint to move the identity and commerce page types from logged to enforced. Teams that license the full database instead of the API add a data-loading step, but skip the network hop entirely once it is in place.
Gateway vendors typically start on the lookup API to prove the integration, then move to an on-prem database license once request volume makes a per-call cost material or a local index materially reduces latency at the enforcement point.
Self-serve API plans run from $99/mo for 90,000 lookups up to $1,997/mo for 2,000,000 lookups, roughly $1.00–$1.10 per 1,000 checks; custom volumes beyond 10M lookups a month are quoted directly. Full database licenses are one-time and perpetual: 10M domains for $14,999, 15M for $24,999, 30M for $49,999, with an optional refresh at 30% of license price per year; 40M+ and OEM redistribution for gateways that want to ship this as a built-in feature are quoted on request. Full detail is on the pricing page.
Fair use on the API tiers is straightforward: it is priced and built for live, per-navigation lookups, not for enumerating the corpus in bulk. A gateway checking each agent request as it happens fits the intended use directly; a nightly job trying to download the entire domain list through the API does not, and that pattern is what the full database license exists for instead.
If your gateway sits in front of an in-house agent platform rather than reselling access to third-party agents, the OEM framing for agent platform vendors may fit your packaging better. Security teams standing up their own enterprise browser or SASE/SSE layer on top of gateway output should read enterprise browser AI agent control and SSE agent traffic policy. For the underlying mechanism, start with agent guardrails and the page-types database. Gateways that also want a category feed for human-facing AI tool traffic can pair this with Web Filtering Database’s 100M+ domain filtering categories.
Two questions come up often enough outside the FAQ format to note here. First, the classification is method-agnostic on the page-type side — a URL is a login page whether requested by GET or POST — while the companion egress rules library is explicitly method-aware for the write surfaces where that distinction matters, such as a wiki edit endpoint that only becomes a write on certain verbs. Second, the lookup itself is tenant-agnostic: it returns the same page-type classification for a given URL no matter who asks, and per-tenant policy differences belong in your own policy layer on top of it, scoped the same way you already scope any other per-customer rule.
One lookup, 28 page types, 40M+ domains. Start with the API docs, or license the database for OEM redistribution.
