Teams keep asking whether an AI agent allowlist replaces robots.txt, or whether robots.txt already covers agent behavior. Neither question has a clean yes, because the two files answer different questions for different audiences. robots.txt is a site publisher's own statement about which automated crawlers may index which paths. An agent allowlist is an operator's own policy about what its own autonomous agent may do once it reaches any of 40 million+ other domains it does not control. You need both, and they are not substitutes for each other.
robots.txt would not have stopped any of the 2026 agent incidents. Several high-profile cases — a Hugging Face breach, a wiki hijack on DseWiki, a JFrog Artifactory covert channel, and third-party account takeovers — all crossed pages that robots.txt files never even mention, like upload endpoints and login forms. Our database and egress rules would have prevented almost all of them.
A file called robots.txt lives at the root of nearly every domain on the internet, and a policy engine that governs agent browsing also decides, per URL, whether to let a request through. From a distance the two look like they occupy the same job: telling automated software where it may go. Close up, they answer different questions, for different audiences, with different enforcement, and conflating them leaves a gap that neither one covers.
robots.txt is written by the operator of the site being visited. It says, in effect, "here is what I, the publisher, am comfortable with automated crawlers indexing." It was designed in 1994 for search-engine spiders deciding what to add to an index, and it has no concept of an autonomous agent that clicks a "start free trial" button, fills out a form, or reaches a checkout on behalf of a third party. Its enforcement is entirely voluntary: a well-behaved crawler reads the file and honors it, and a crawler that ignores it faces no technical barrier at all, only whatever consequence the site operator can impose after the fact.
An agent allowlist is written by the operator of the agent, not the operator of the site. It says, in effect, "here is what I, the company running this agent, will let it do once it reaches any domain on the open internet, whether that domain has an opinion about agents or not." Its enforcement sits inside your own gateway or framework, checking every outbound request before it leaves, regardless of what the destination site has or has not published. It does not depend on the other 40 million domains cooperating, because most of them never will.
That difference in authorship is the whole story. A site publisher cannot see your agent's task, your company's risk tolerance, or the fact that this particular agent should never touch a login form under any circumstances. Your own policy engine cannot see, and should not need to see, every site's editorial preference about search indexing. Each one is the right tool for the party who wrote it, and neither one was built to do the other's job.
A plain-text file a site publishes at /robots.txt, listing Disallow and Allow paths per user-agent, addressed to crawlers deciding what to fetch and index. It is a request, not a lock: nothing on the wire stops a client from fetching a disallowed path, and the file says nothing about what happens once a page loads — it cannot distinguish a search-indexing GET from an authenticated form submission, because both look the same to a text file matched against a URL path.
A ruleset the company running an autonomous agent applies to every URL that agent is about to request, anywhere on the internet, based on the page's verified type — login, checkout, pricing, docs, and 24 more. It is enforced at your gateway or framework, before the request leaves, independent of whether the destination site published any crawler guidance at all. It answers "may my agent do this," not "should a search engine index this."
This is the centerpiece comparison, because most confusion about "do we still need an allowlist if we already respect robots.txt" collapses once these dimensions are laid out next to each other.
| Dimension | robots.txt | AI Agent Allowlist / page-type policy |
|---|---|---|
| Who writes the rule | The site being visited, for its own domain only | The company operating the agent, applied to every domain it visits |
| Who it is addressed to | Search-indexing crawlers, historically | Your own agent's browsing decisions, at request time |
| Enforcement mechanism | Voluntary; the crawler chooses to honor it | Enforced in your gateway or framework before the request is sent |
| Granularity | URL path patterns per user-agent string | 28 verified page types per domain plus ~40 method-aware egress patterns |
| Coverage if the site says nothing | No file, no rule — default is usually "crawl everything" | Default-deny for unclassified destinations, regardless of what the site publishes |
| Covers login, checkout, signup specifically? | Only if that site's author thought to list them, and most never do | Yes, by page type, on every classified domain, without per-site authoring |
| Distinguishes GET from a write action | No — a path match is a path match regardless of method | Yes — egress rules key on write methods (POST, PUT, MKCOL, and so on) |
| Where the audit trail lives | Nowhere; robots.txt keeps no log of who obeyed it | In your own policy engine: every allow, deny, and flag decision logged per request |
| Updated by | The destination site, on its own schedule | Your own policy plus a database refreshed on your license terms |
| What it protects | The publisher's preference about search-index inclusion | Your company's exposure from what its own agent does on someone else's site |
Note the row that matters most in practice: robots.txt has no reliable way to name a login page, a checkout, or an upload form, because it was never designed to. A site's own author would have to specifically decide to list Disallow: /login, and the overwhelming majority never do — not because they welcome agent logins, but because robots.txt was built for search indexing, and search engines were never going to index a login form anyway. Silence in robots.txt is not permission. It is simply a file that was never asked the question.
The two are not in tension. A well-built agent stack respects robots.txt when it is doing anything resembling crawling or indexing — that remains good internet citizenship and, for search-style agents, keeps you off site operators' block lists. The same stack also runs a page-type policy check before every request, because robots.txt was never going to name a login page for you, and no amount of respecting it changes that.
A vendor-research agent is told to pull pricing and product pages from five SaaS vendors. Here is what happens when its gateway checks both robots.txt and the page-type policy on the same five requests.
| Request | robots.txt says | Page-type policy says | Result |
|---|---|---|---|
| /pricing on Vendor A | Not mentioned | pricing → allow | allowed |
| /blog/* on Vendor B | Disallow: /blog/drafts/ only | blog → allow | allowed |
| /login on Vendor C | Not mentioned — most sites never list it | login → deny | denied by policy, robots.txt silent |
| /app/signup on Vendor D | No robots.txt file published at all | signup → deny | denied by policy, no robots.txt to consult |
| /docs/api on Vendor E | Allow: /docs/ | documentation → allow | allowed |
Three of the five requests would have looked identical with or without a page-type policy, because robots.txt happened to agree or say nothing relevant. The two that matter are the login and signup requests: robots.txt was silent on both, either because the site's author never listed them or because the site published no robots.txt file whatsoever. A stack that only checks robots.txt treats silence as "nothing to stop me" and proceeds. A stack that also runs page-type policy treats an unclassified or dangerous page type as deny by default, which is the entire reason default-deny exists as its own layer, independent of what any individual site chose to publish.
Over the past two years, many sites added new user-agent tokens to their robots.txt files aimed specifically at AI companies' training and retrieval crawlers — disallowing a named bot from harvesting content for model training or for a retrieval index. This is a real and useful development, and it is worth honoring if your organization runs that kind of crawler. It is also a different problem from the one an agent allowlist solves, and it is easy to conflate the two because both involve the letters "AI" and the same file.
A training-data or retrieval crawler is reading content to build or feed a model, and the site's concern is about how its content gets used downstream, not about what any single request does at the moment it happens. An autonomous agent acting on a user's or a company's behalf is doing something else entirely: clicking, filling forms, and navigating in real time, on a mission that has nothing to do with training data. A site's robots.txt entry disallowing a named training crawler says nothing about whether an unrelated agent, browsing under a generic user-agent string or a browser-automation stack, should be allowed to reach that same site's login form or checkout page. The two concerns do not overlap enough for one file to answer both.
This is also why page-type policy cannot be replaced by "just add more AI-specific tokens to robots.txt" as a fix. Even a robots.txt file with a dozen named AI-crawler disallow lines still has nothing to say about a login page on a site that never anticipated agent traffic at all, which describes the overwhelming majority of the 40 million+ domains an operator's agents might encounter. The token-based approach scales by the number of named crawlers a site's author remembered to list; page-type policy scales by page type, which is a fixed and small vocabulary that applies whether or not the destination site has heard of your agent.
If your agent does anything resembling crawling or indexing, continue honoring robots.txt for that behavior. It remains the correct signal for that narrow job.
Check every outbound request against a verified page-type map before it leaves, regardless of what robots.txt says or whether it exists on that domain.
A domain with no robots.txt and no page-type classification is not a green light. Unclassified destinations should fail closed, not open.
robots.txt keeps no record of compliance. Your audit trail for what the agent actually did belongs in the policy engine that made the allow/deny call.
Notice the ordering: robots.txt only enters the decision at all when the agent is doing something crawler-like, and even then it never overrides a deny from the page-type layer. A site cannot use its robots.txt file to grant your agent permission to submit a form, because robots.txt has no vocabulary for permission of that kind — it only ever spoke about indexing.
The page-type check is a single HTTP request your gateway or framework makes before letting the agent navigate, independent of any robots.txt fetch you may also be doing:
High-volume or latency-sensitive deployments typically license the full database rather than call the API per request — a local lookup with no external dependency. Request and response examples live in the API docs; plan and license pricing is on the pricing page.
For the same read-vs-write, site-vs-operator distinction applied to what humans install and run rather than what agents browse, see AI Tools Blocklist, the sibling product covering 20,000+ AI-tool domains by risk category.
28 page types, 40M+ domains, verified URLs. Start with the free sample, then pick a lookup plan or a database tier.
