An AI agent allowlist is a policy data source that tells a system, before an autonomous agent requests a URL, whether that specific page is one the agent is permitted to visit. It is not a firewall, a content filter for humans, or a list of "good" websites — it operates at the level of individual pages, keyed to what each page actually lets someone (or something) do. This page explains the concept from first principles: what problem it solves, how it is typically enforced, and where it sits next to related ideas like blocklists and zero trust.
An AI agent allowlist is a database that maps, for each of many millions of domains, which specific pages exist for which purpose — a login form, a pricing page, a checkout flow, a documentation hub — so that an autonomous software agent's request to visit a URL can be checked against a policy before the request happens, rather than only monitored after the fact. Where a traditional web allowlist might say "this domain is approved," an agent allowlist says "this domain is approved for research reads, but its login and checkout pages are not," because the two carry entirely different risk even on the same site.
That page-level distinction is the entire reason the concept exists. An AI agent tasked with researching a vendor needs to reach that vendor's pricing and documentation pages freely; the same agent reaching that vendor's login page, cart, or account-signup form is very likely a mistake, a misconfiguration, or a hijacked task — not a legitimate outcome of "doing research." A domain-level allowlist cannot express that difference. A page-type-aware one can.
Before agent allowlists, the closest available tool was ordinary domain or category-based web filtering — the kind built for blocking humans from reaching malware or adult content. That tooling answers "is this domain safe," a question built around whole sites having roughly one character. It breaks down immediately for agentic browsing.
Take a single, ordinary example: a well-known SaaS company's marketing site. Its blog, documentation, and pricing pages are exactly what a research agent should read freely; its customer login and billing pages are exactly what that same agent should never touch unattended. Both live on the same domain, often the same subdomain, sometimes one link apart. A policy that operates at the domain level has exactly two options — allow the whole site, including the login form, or block the whole site, including the pricing page a legitimate research task actually needs. Neither option matches what the task requires, and this single-domain problem repeats across every domain an agent fleet might ever touch, at a scale no manual review process can keep up with.
Our companion piece, why domain blocking fails agents, goes deeper into this specific failure mode with worked examples. The short version: the unit of policy has to be the page, not the site, and knowing which page is which — at the scale of tens of millions of domains — is what an agent allowlist actually provides.
The same reasoning rules out guessing at URLs as a shortcut. A policy engine that tries "/login" or "/checkout" on every domain it encounters will be wrong often enough to be unusable: real login pages live on separate identity subdomains, behind locale prefixes, or on third-party identity providers entirely, and a guess that returns a 404 or the wrong page is worse than no answer at all if the policy silently treats it as "not present." Verifying each page type from the domain's actual live link structure, rather than guessing a path pattern, is what lets an allowlist make a page-level decision it can stand behind.
Until fairly recently, most software that touched the web on an organization's behalf followed a fixed, human-written path: a specific API call to a specific endpoint, a scraper targeting a specific selector, a script hitting a specific list of URLs someone typed in advance. None of that needed a page-type allowlist, because a human had already decided, once, exactly where the code was allowed to go.
Autonomous agents built around large language models broke that assumption. An agent given a goal — "research this vendor," "find the current pricing for this tool," "check whether this service is compatible with our stack" — decides its own navigation path at run time, often across domains nobody on the team has visited before, sometimes following links several hops deep from wherever it started. That is precisely the capability that makes these agents useful, and precisely the capability that makes a fixed, human-curated list of approved URLs unworkable: the agent's value depends on reaching pages nobody anticipated, and a hand-maintained list cannot anticipate them either.
An AI agent allowlist resolves that tension by moving the decision from "which specific URLs did we pre-approve" to "given any URL the agent proposes, what kind of page is it, and does our policy permit that kind." The agent keeps its ability to go anywhere its task requires; the organization keeps a deterministic answer about the sensitive categories of pages — logins, payments, account changes — it never wants an unattended agent reaching, no matter which domain those pages happen to live on.
In practice, "an AI agent allowlist" is usually implemented as four complementary layers, checked in sequence by a gateway, proxy, or in-process hook sitting between the agent and the network. Each layer covers a gap the others do not.
Around 60 specific hosts — cloud metadata endpoints, registries, tunnel services — that are denied by host identity alone, regardless of which page-type database entry they might otherwise match. See the high-value host list.
For each of 40 million-plus domains, up to 28 verified page-type URLs — login, pricing, checkout, documentation and more — so a policy can allow or deny by what a specific page actually is. See the full schema.
Roughly 40 method-aware rules matching patterns like wiki-edit endpoints, WebDAV paths, or plugin-install URLs, which apply even to domains outside the core database. See the egress rules library.
An unrecognized destination is refused rather than allowed by absence of a record. This is what makes the system an allowlist in the strict sense, rather than a blocklist with extra steps — see the comparison on the sibling page below.
A single domain's record in the database carries more than page-type URLs, because a full policy decision often depends on more than "what kind of page is this."
This matters because "should an agent visit this page" is sometimes a content-risk question (is this domain in a category your organization treats as off-limits) as much as a page-type question (is this a login form). The two taxonomies — content category and page type — are complementary, and a mature policy uses both. The free sample CSV includes all of these columns across 100 real domains so you can see the shape of a record directly.
The word "allowlist" carries a specific technical commitment: unless a destination is explicitly known and permitted, it is denied. That is a different default from a blocklist, which permits everything except a known-bad list — and the default is the whole difference in a system that has to make a decision about a domain nobody has ever explicitly evaluated before.
In practice, the four-layer model above is a hybrid: the high-value host list and the egress rules function like a curated blocklist for known-dangerous shapes, layered on top of a true default-deny allowlist backstop. That nuance, and when each pure model actually wins on its own, is the subject of our dedicated comparison.
Read the full allowlist vs. blocklist comparisonThe allowlist itself is data, not a finished policy — a gateway, proxy, or SDK guardrail hook applies rules on top of it. Some common shapes those rules take:
Two groups tend to adopt this concept for different reasons. Security and platform teams at organizations already running agents care about it as a control — a deterministic answer to "what did our agents actually touch," useful for audits and incident response as much as prevention. Agent-platform vendors and AI gateway builders care about it as a product ingredient: a guarantee they can offer their own customers ("this agent will never reach a login page") is easier to sell, and easier to keep, when it rests on verified data rather than a prompt instruction the model might not follow under pressure.
The term is new enough that it gets conflated with adjacent, older ideas. Separating them makes the definition at the top of this page more precise.
It is not a content filter for humans. Tools like our sibling product, the AI Tools Blocklist, exist to control which AI-related websites a human employee can reach from a corporate network. An agent allowlist runs in the opposite direction: it controls which pages a piece of autonomous software can reach on the human organization's behalf. The two are complementary rather than substitutes, and a mature security posture typically has both.
It is not a substitute for prompt-level safety work. Instructing a model not to visit certain kinds of pages, or filtering the content an agent reads before it reaches the model, both reduce how often an agent decides to go somewhere it should not. An allowlist is what still holds when that upstream reasoning is wrong, manipulated, or simply mistaken — it evaluates the destination URL itself, independent of why the agent chose it. See our dedicated piece on defending against injected redirects for the detailed version of this argument.
It is not a one-time list someone maintains by hand. A hand-curated list of a few hundred approved domains is a reasonable starting point for a narrow pilot, but it does not scale to open-ended research tasks, and it drifts out of date the moment sites restructure their navigation. A database built by systematically traversing and re-verifying millions of domains is what makes the approach viable past a pilot.
It is not the same as robots.txt or a sitemap. Those describe what a site's own owner wants crawled, from the site's perspective, and say nothing about identity or transaction risk. An agent allowlist encodes the operator's own risk-based policy about what its agents should be permitted to do, informed by verified facts about what each page is, independent of what the target site's owner intended for a crawler.
Several high-profile 2026 incidents involved AI agents escaping their intended scope or being hijacked — a Hugging Face dataset-upload breach, a wiki hijack that produced roughly 15,000 unauthorized edits, a covert channel through JFrog Artifactory, and third-party account takeovers. Our analysis shows a page-type allowlist and its egress rules would have denied nearly every entry point pre-request.
The 2026 agent incidents, prevented Read the Hugging Face breach caseThe honest fine print — the same two assumptions we publish, plus two operational ones
Default-deny vs. default-allow-with-exceptions, and when each wins.
The full taxonomy, with a real-world example of each.
The product-level view of the four-layer enforcement model.
The companion 100M+ domain content-filtering dataset.
Download the sample CSV or browse the full page-type schema to see exactly what an agent allowlist record looks like.
