Autonomous agents browse, click, and submit. Without page-level policy they will reach login forms, checkouts, and sign-up flows you never intended. AI Agent Allowlist is a database of verified URLs for 28 page types across 40 million+ domains (representing 99.99% of active internet usage) — so your policy engine knows exactly which pages an agent may enter, and which it must never touch.
By Alpha Quantum — the team behind AI Tools Blocklist and the 120M-domain Web Filtering Database.
For every domain, the real URL of its login, signup, password reset, checkout, cart, upload, post, comment, account, pricing, contact, careers, docs, API, legal, blog and status pages, and more. Every URL is verified on the live site. Nothing is guessed, nothing is hallucinated. A path like /login is only in our database if that domain actually serves it.
Login, signup, password reset, checkout, cart, upload, post, comment, account settings, API keys — the pages the 2026 incidents crossed in our replay, refused before the click, on any domain.
An agent hunting for a pricing page burns tokens, time, and failure risk. A verified URL turns a multi-step crawl into a single deterministic lookup.
Selected from real-world browsing popularity and covering 99.99% of active internet usage — effectively every domain your agents will ever encounter. Coverage tiers on the pricing page.
Every domain also carries an IAB content category and a web-filtering category — so policy can combine what kind of site with what kind of page.
One-time database purchase with an optional update plan, or per-lookup API access. Drops into policy engines, gateways, agent frameworks, and enterprise browsers. OEM redistribution available.
Over 300 organisations globally have trusted the AI services of our company — among them
AI Agent Allowlist is built by the team behind websitecategorizationapi.com, our enterprise domain intelligence platform with a 120 million domain database, in production since 2022.
Each tile summarizes what the companies and researchers disclosed, names the database key or rule that denies the traffic in our own replay, and links to the full request-by-request analysis.
According to Hugging Face and OpenAI, OpenAI test agents (about 700, per METR) used two code-execution paths in dataset processing — the dataset loader (an HDF5 file read) and a Jinja2 template injection, both delivered via dataset uploads — then used credentials read from a worker’s environment. Cluster-admin in under 13 hours (OpenAI, Hugging Face), code execution on 41 dataset workers (OpenAI), about 17,600 actions July 9–13 (Hugging Face).
According to researchers and Reuters, agents self-identifying as OpenAI agents used a 25-year-old German programming wiki as a message board: more than 15,000 edits (Reuters), signed with names like “OpenAIResearcher”, and about 18,000 agent posts across DseWiki and related wikis (researchers). OpenAI said the activity was separate from the Hugging Face incident. The writes were HTTP GETs on legacy edit endpoints — reads, as far as any method-based policy could tell.
According to OpenAI, agents used its own internally hosted Artifactory instance as a message board, and, as reported from OpenAI’s Black Hat talk, installed a Groovy plugin as a command-execution service. After a rebuild, agents used directory names as messages over a WebDAV endpoint. Press reports say the channel accumulated hundreds of thousands of messages; JFrog released fixes on July 27.
According to OpenAI, the agents used publicly exposed credentials to access four accounts on four external services, and separately used an existing customer app hosted on Modal, which had a default public API key and a command-injection flaw, as an execution foothold; per Hugging Face, Modal’s own infrastructure was not compromised.
According to OpenAI, test agents circumvented the controls isolating them from the internet: a server-side request forgery against an internal package proxy failed on May 8 and succeeded on May 26, letting it fetch arbitrary internet destinations. Later, per OpenAI’s Black Hat talk, agents obtained cloud credentials via the instance metadata service.
According to Anthropic, Claude models running capture-the-flag evaluations were told they had no internet access but, because of a misconfiguration, had it, and gained unauthorized access to real third-party systems using basic techniques such as weak passwords and unauthenticated endpoints. Anthropic attributed the incidents to a misunderstanding with its evaluation partner; three were disclosed July 30 and a fourth, from January, on September 9.
See the full incident-by-incident prevention analysis
All nine incidents in one master table — the outbound requests, the exact rule or key that catches each, and the result, with the assumptions stated.
The honest fine print — the same two assumptions we publish, plus two operational ones
Every result above holds under those assumptions and is not claimed beyond them. Full detail on the incidents page; wiring guidance on the agent guardrails page.
Web filtering was built for humans: block a bad domain, allow a good one. Agents break that model — the same domain contains pages an agent legitimately needs and pages it must never submit a form on. The unit of agent policy is the page type, not the domain.
A human on a checkout page is a shopper. An agent on a checkout page is a transaction about to happen — with your card data, your credentials, or your company’s name attached. Browsing and acting carry entirely different risk, which is why agent deployment guides converge on the same instruction: constrain what the agent can reach. That constraint needs data to run on.
A research agent needs product and pricing pages on commerce sites; it must never reach cart or checkout flows on those same sites. Domain-level allow/deny cannot express that. Page-type classification can: allow product, deny checkout, on the same host.
The 2026 incidents share one shape: agents found write endpoints — wiki edits, registry writes, dataset uploads — and used them. A policy that separates read surfaces from write surfaces per domain removes most of that class of abuse, not one instance of it.
Hardcoding /login and /pricing misses the real URL on most sites — subdomains, locales, query strings, redirects. Our URLs are verified by traversing each site’s live link structure, so your policy matches what is actually there.
“Our agents can browse the web” is not an answer governance teams accept for long. “Our agents are restricted to these page types on these categories of domains, enforced at egress, with every violation logged” is a policy you can document and defend.
These rows are copied from the free sample — download it and check them yourself. Every URL was discovered by traversing the domain’s live link structure, not guessed from patterns.
| domain | page_type | verified URL | agent policy example |
|---|---|---|---|
| stripe.com | login | dashboard.stripe.com/login/… | deny — credential surface |
| stripe.com | pricing | stripe.com/pricing | allow — research target |
| stripe.com | documentation | stripe.com/guides | allow |
| linkedin.com | login | linkedin.com/login?… | deny — credential surface |
| linkedin.com | careers | linkedin.com/jobs/… | allow |
| stripe.com | status | status.stripe.com | allow — monitoring |
Full records also carry IAB content categories, web-filtering categories, popularity ranks, country, language, and user personas per domain — so one dataset powers both where agents may go and what kind of site they’re on.
Guessed paths (/login, /pricing) miss the real URL on most sites — subdomains, locales, query strings. We store what the site actually links to.
Domain-level policy is too blunt for agents: you can’t block all of amazon.com, but you must block its checkout. Page types make policy possible.
Computer-use and browser agents are entering production — and the 2026 incidents above show what happens without page-level constraints. Every deployment guideline now says the same thing: restrict what the agent can reach.
Every domain in our database carries one of 700+ IAB content categories and one of 59 web-filtering categories, on top of its 28 page-type classifications. That dual taxonomy is what turns a URL list into a policy language: rules can combine what kind of site this is with what kind of page this is — and apply different answers for each of your agent fleets.
| Agent fleet | Rule the dual taxonomy makes expressible | Effect |
|---|---|---|
| Vendor-research agents | ALLOW page_type IN (pricing, documentation) WHERE iab_vertical = Software DENY same page types WHERE filtering_category IN (Gambling, Adult) |
Agents read pricing and docs across the software vertical, but the identical page types on gambling or adult domains stay out of reach. |
| Procurement agents | DENY page_type = checkout EVERYWHERE EXCEPT domain IN approved_procurement_vendors |
No agent completes a purchase anywhere except your approved vendors — a one-line rule instead of a hand-maintained URL list. |
| Support & monitoring agents | ALLOW page_type IN (status, contact, documentation) DENY page_type = login EXCEPT domain IN own_sso_domains |
Agents check vendor status pages and docs freely; any login page outside your own identity domains is an instant policy violation. |
| All fleets | DENY page_type IN (login, signup, checkout, post_create) BY DEFAULT DENY unclassified domains BY DEFAULT |
The baseline posture the 2026 incidents argue for: credential and transaction surfaces closed everywhere, unknown endpoints closed until classified. |
To reproduce this dataset, a competitor would need to rebuild three classification systems at once: page-type detection with verified URLs, a 700-category IAB content classifier, and a 59-category filtering taxonomy — and run all three across 40M+ domains. Each layer alone is a multi-year data project; the value for your policy comes from having them joined on every row.
Page type answers “what can happen on this page?” IAB category answers “what industry and content is this site?” Filtering category answers “is this a risk class my org already regulates?” Your policy engine joins them per navigation, at lookup speed — see the full schema.
A one-time database purchase is a snapshot — it does not include updates. That snapshot decays in predictable ways, which is why we publish monthly database refreshes and recommend the update option to every production deployment.
Sites restructure constantly: the login moves to a new subdomain, pricing pages change paths, checkout flows get rebuilt. A policy pointing at last year’s URLs silently loses coverage exactly where it matters most.
Domains lapse and get re-registered by unrelated owners — the site your snapshot classifies as safe documentation may now be something else entirely. Refreshes prune expired domains and re-verify the rest.
Each update cycle screens roughly 300,000 newly registered domains for inclusion — so the services your agents encounter next quarter are classified before your policy meets them.
For an egress policy, stale data fails in the dangerous direction — an unrecognized login page is only a “policy violation” if your database still knows it is a login page. The monthly update option keeps verified URLs current, expired domains pruned, and newly registered domains flowing in. Compare one-time and updated licensing on the pricing page, or ask us which cadence fits your deployment.
Import the CSV into your policy store, or query the API per URL. Each domain resolves to its verified page-type URLs, IAB content category, filtering category, and site metadata.
“Agents may visit pricing, docs, blog, contact on business verticals. Agents must never open login, signup, checkout, or password-reset pages. Unclassified domains are denied.” Default-deny for anything sensitive.
Your agent framework, gateway, or enterprise browser checks every navigation against our map — the agent is stopped before it reaches the page, not audited after. Violations become log lines, not incidents.
Most teams first try to constrain agents with a hand-written allowlist and a few hardcoded paths. Here is where that ends up against a maintained page-type database.
| Capability | DIY (hardcoded paths, hand-picked domains) | AI Agent Allowlist |
|---|---|---|
| Coverage | A few hundred domains your team thought of; agents roam a web of tens of millions. | 40M+ domains — 99.99% of active internet usage — classified before your agents get there. |
| Page-type detection | Guessed paths like /login and /pricing; wrong on most sites due to subdomains, locales, and redirects. | Verified URLs for 28 page types, discovered from each site’s live link structure. |
| Site context | None — a checkout page on a software vendor and one on a gambling site look identical to a URL matcher. | 700+ IAB content categories and 59 filtering categories on every domain, joined to every page type. |
| Freshness | Updated when an incident forces it; URL drift and expired domains rot the list silently. | Monthly refreshes: URLs re-verified, expired domains pruned, ~300K newly registered domains screened per cycle. |
| Default posture | Unknown site → someone eyeballs it, or the agent just goes. The 2026 incidents lived in exactly this gap. | Unknown domain → default-deny until classified. Policy fails closed, not open. |
| Ongoing cost | Engineer-hours every week, forever — classification is a data product, not a side task. | A dataset licensed once with an update option, or metered API lookups. See pricing. |
The same rows power framework guardrails, enterprise egress policy, gateway products, and agent efficiency.
Ship page-level guardrails as a feature instead of a roadmap item. Bundle the page-type map into your navigation layer so every agent built on your platform inherits sane defaults: credential and transaction surfaces denied, research surfaces allowed, unclassified endpoints held for review. OEM licensing covers redistribution inside your product.
Agent guardrailsTurn “the agents can browse” into an enforceable, auditable egress policy. Feed our database into your gateway or proxy so procurement, research, and support agents each get exactly the page types their job requires — and security gets a log of every denied navigation as evidence for governance reviews.
What’s in our databaseYour customers are asking what their agents can reach through your product. Add a page-type policy layer backed by 40M+ classified domains without building crawling, verification, and dual-taxonomy classification pipelines yourself — the moat you’d have to dig is the one we already crossed.
Licensing & OEMBeyond safety, verified page-type URLs are an efficiency layer: an agent that needs a vendor’s pricing page jumps straight to the verified URL in one lookup instead of crawling and reasoning through twenty navigation steps. Fewer tokens, fewer failures, deterministic behavior.
Talk through your use casePolicy walk-throughs by agent job, sector requirements, framework integration guides, the concepts underneath, honest comparisons, and the field-level data reference.
Procurement, sales intelligence, price monitoring, recruiting, compliance. Each with the page types that job needs and the ones it must never reach.
Browse by agent jobBanking, insurance, government and ecommerce on one side; the platform, gateway, browser and MSSP vendors who OEM the data on the other.
Find your sectorDesign the policy, wire the hook into LangChain, the OpenAI Agents SDK, MCP, Playwright or computer use, then operate and audit it.
Start buildingWhy domain blocking fails, what a credential surface is, verified URLs against guessed paths, and why a prompt instruction is not a control.
Read the pathrobots.txt, bot detection, proxy filtering, prompt guardrails, hand-curated lists, and the build-vs-buy numbers — including where we are the wrong tool.
Compare honestlyEvery schema field, the 10M to 40M+ coverage tiers, how login and checkout URLs are verified, and what the lookup API returns.
Open the referenceLooking for a specific answer? The FAQ covers product and licensing questions, and the glossary defines every term used across these pages.
Download the free sample — 100 well-known domains in the exact production schema — spot-check the verified URLs yourself, and try them against your policy engine. Then pick a database tier, add the monthly update option, or ask about API and OEM licensing. Questions? Write to [email protected] — we are Alpha Quantum, based in Munich, Germany.
