To build this database we traversed the live link structure of each domain and individually analyzed over 10 billion links through a multi-step AI classification pipeline. The result is not a list of guessed paths — it is the actual URL a site links to for each of 28 page types, combined with 700+ IAB content categories and a 59-category filtering taxonomy, across a repository of more than 40 million domains representing 99.99% of active internet usage.
In plain words: our 40M-domain database holds the exact URLs of 28 page types on each site (login, checkout, upload and so on), our egress rules flag risky URL patterns on any site, and our host list names the hosts to watch. Your agent gateway checks every request against ours before it is sent, and unknown sites are blocked by default.
These classifications, replayed against 2026. See how, in our own replay of the public record, our database + egress rules deny the web-facing steps of the documented 2026 agent incidents — including the Hugging Face incident.
Over 300 organisations globally have trusted the AI services of our company — among them
AI Agent Allowlist is built by the team behind websitecategorizationapi.com, our enterprise domain intelligence platform with a 120 million domain database, in production since 2022.
Domain-level policy is too blunt for agentic browsing. You cannot block all of a retailer’s site, but you must block its checkout; you want an agent reading a vendor’s documentation, but never standing in front of its login form. That distinction only works if you know, per domain, where those pages actually live. For each domain in the database we store up to 28 page-type URLs, verified from the site’s live link structure. Availability depends on each site — not every domain has all 28. The types are grouped below by what they let an agent do; the deny/flag/allow results are a typical starting policy, and you define your own rules on top of the raw classifications. The eight action types — pages where an agent does something (creates an account, submits a form, spends money, publishes content) rather than just reads: signup, password_reset, cart, checkout, upload, post_create, comment, subscribe — are shipping with the current reclassification cycle; records gain them as domains are re-verified.
Why verified URLs rather than guessed paths? Because /login is wrong more often than it is right. Real login pages live on subdomains (dashboard.stripe.com/login), behind locale prefixes, or on separate identity providers entirely. Every URL in this database was discovered by traversing what the domain actually links to — which is also why the same method can tell you a page type genuinely does not exist on a site, itself a useful policy signal.
The per-domain database is built only from pages a site links from its own homepage structure — navigation, footers, sitemaps, internal cross-references. It never scans for admin or login paths, never probes URLs a site does not advertise, and never sends form submissions. That is why it is safe to run across tens of millions of domains: to the sites it classifies, it does not behave like a scanner.
URL-level coverage of admin panels and write endpoints comes from two companion products instead: the rules file — URL-pattern rules evaluated on the exact URL before the request, on any domain including unclassified ones, method-aware so a GET on a wiki edit URL is a read while a POST is a write — and the host list, a curated list of dangerous infrastructure hosts: cloud consoles, package registries, paste sites, webhook and tunnel sinks, mail senders and cloud metadata endpoints. Curated, not crawled — so the layer that guards the most sensitive surfaces never had to touch them.
Identifying up to 28 page types per domain is not a crawl-and-guess exercise. Every link our traversal discovered was individually fetched, its content and structure examined, and its page type assigned by a multi-stage AI classification pipeline — more than 10 billion links in total.
Knowing where the pricing page is answers half the question. The other half is what kind of site the agent is standing on. Every domain in the database therefore carries three classification layers that interlock: the 28 page-type URLs, one of 700+ IAB content categories, and one of 59 web-filtering categories.
The behavioral layer: which URLs on this domain are credential surfaces, transaction flows, or safe reading targets. This is the axis your enforcement engine checks on every navigation.
The vertical layer: IAB v2 and v3 tiers 1–4 tell you whether the domain is a bank, a SaaS vendor, a hospital, a school, or a news outlet — so policy can differ by industry, not just by page.
The risk layer: adult, gambling, weapons, illegal content, proxies and other block-worthy classes from our web-filtering taxonomy — the domains where no page type should be reachable at all.
The combination is what makes genuinely granular agent policy possible. A page type alone cannot express “docs are fine on software vendors but not on gambling sites.” A content category alone cannot express “read the catalog but never the checkout.” Together, they can express both — and everything between:
| example policy | page-type axis | taxonomy axis | effect |
|---|---|---|---|
| Research agents in the software vertical | docs, pricing | IAB: Software & Technology | allow |
| Same page types, hostile vertical | docs, pricing | Filtering: Gambling | deny — category overrides |
| Procurement agent, approved vendors | cart, checkout | domain on internal approved-vendor list | allow for this agent role only |
| All other checkout pages, any vertical | cart, checkout | any | deny by default |
| Finance-vertical caution | any interactive type | IAB: Personal Finance / Banking | require human approval |
| Risk-category hard stop | all 28 types | Filtering: adult, weapons, proxy/anonymizer | deny domain entirely |
| Monitoring agents, news industry | blog/news, press | IAB: News & Media | allow, read-only |
Reproducing the page-type layer means traversing the live link structure of tens of millions of domains and individually classifying billions of discovered links — our pipeline analyzed over 10 billion — not pattern-matching paths, which fails on subdomains, locales and identity providers.
The content layer comes from the same classification stack that powers our commercial website-categorization products: IAB v2 and v3 down to tier 4, trained and refined over years across more than 40 million domains. That classifier is a product in its own right — here it ships as one column.
The risk layer reuses the 59-category taxonomy behind our web-filtering database, built for network security vendors where a miscategorized domain is a support ticket. Assembling all three layers — and keeping them consistent per domain — is the real barrier, and the reason teams license the data instead of rebuilding it.
The full repository covers more than 40 million domains, selected and ranked from real-world browsing popularity data. Licensed tiers are cut from this repository by popularity rank:
Custom slices — by industry, geography, language or page-type availability — can be built from the complete repository on request via contact.
Internet traffic is extremely concentrated — the long tail of parked and dormant domains contributes almost nothing to what an agent will actually encounter.
In practice, an agent navigating the web checks the database and gets an answer — not a miss.
Industry, geography, language or page-type availability — cut to order from the complete 40M+ repository.
A domain outside the repository is itself a signal: default-deny until classified, so policy fails closed, not open.
Page types are the core, but policy needs context: what kind of site is this, how popular, where, in what language, for whom? Each record ships with all of the following fields.
One row per domain, page types as delimited type=URL pairs. Loads into any warehouse, policy store or spreadsheet. The default for database tiers.
Nested documents per domain — the format the lookup API returns, and available as JSON Lines for bulk delivery when you prefer objects to columns.
An importable SQL dump with indexed domain lookups, for teams that want to stand up an internal lookup service without an ETL step.
Want to inspect the schema before talking to anyone? The free 100-domain sample contains full records for well-known domains — same columns, same format as a licensed delivery.
Researchers: aggregate statistics from this database (page-type prevalence across 40.8 million domains, by language) are free under CC BY 4.0 in the AI Agent Page-Type Statistics dataset.
Two ways to consume the data: per-lookup API access for teams that want a hosted endpoint, or a full database license for teams that need the data inside their own infrastructure with no per-query dependency. Full details, checkout and OEM terms are on the pricing page.
Hosted JSON endpoint: query any domain, get page types plus both taxonomies. Plans scale by monthly request volume.
API plans →One-time perpetual licenses of the full schema, 10M ($14,999) to 30M domains ($49,999), for on-prem policy engines. OEM licensing for redistribution.
License details →The most-visited 10 million domains — the traffic core. Full schema, delivered as CSV, JSON or SQL dump.
License details →Extends coverage into regional and niche properties — the balance point for most production agent deployments.
License details →Near-complete coverage of the active web for platforms that cannot afford a miss on any navigation.
License details →Redistribution inside your product, custom slices from the 40M+ repository, or bespoke field sets — priced per case.
Talk to us →Every database tier is available as a one-time purchase or with monthly refreshes — and the difference matters more than it looks. Read on.
A page-type database is a snapshot of a moving target. A one-time purchase is exactly that: your copy is accurate as of the delivery date, and it receives no updates of any kind. From day one, the web drifts away from it. The monthly refresh option keeps your copy aligned with what is actually live.
Sites restructure constantly: logins migrate to identity subdomains, checkouts get rebuilt on new platforms, docs move to new hosts. A verified URL from last year can be a redirect — or a 404 — today, and a policy keyed to it silently stops matching.
Domains expire and are re-registered by different owners every day. A domain that was a harmless blog at purchase time can later serve entirely different content under a new operator — a formerly safe entry in a frozen allowlist can become hostile without a single byte of your copy changing.
Each update cycle we screen on the order of 300,000 newly registered domains for inclusion, so refreshed copies pick up the new sites your agents will inevitably be pointed at. A one-time copy never learns they exist.
Pay once, own the snapshot. Frozen at purchase-date accuracy.
Same tiers, plus a fresh delivery every quarter. The option we recommend for anything in production.
If the database drives an enforcement decision — what an agent may click, on behalf of your users — freshness is part of the safety property, not a convenience. One-time licensing exists for evaluations, research and static analyses; for a live guardrail, take the refresh. Refresh pricing per tier is on the pricing page.
Check every proposed navigation against the page-type map before the browser tool executes it — the agent is stopped before the click, not audited after. See agent guardrails for concrete integration patterns.
Load the database into the proxy or browser policy engine that already mediates agent traffic, and enforce page-type rules org-wide without touching individual agents.
Agent platforms and AI-security vendors ship page-level guardrails as a feature by embedding the data under a redistribution license — contact us for terms.
By traversing the live link structure of each domain and running discovered links through a multi-step AI classification pipeline — over 10 billion links individually analyzed in total. We store the URL the site itself links to for each page type, which is why records capture subdomain logins, locale-prefixed paths and third-party identity providers that path-guessing (“try /login”) misses. When a page type does not exist on a domain, it is recorded as absent rather than guessed.
In addition to the 28 page-type classifications, every domain carries one of 700+ IAB content categories (v2 and v3, tiers 1–4) and one of 59 web-filtering categories. Page types tell you what a URL does; the taxonomies tell you what kind of site it belongs to. Combined, they support policies like “allow docs and pricing in the Software vertical, deny the same page types on Gambling domains” or “deny all checkout pages except approved procurement vendors.”
Not yet in every row. The eight action types — signup, password_reset, cart, checkout, upload, post_create, comment and subscribe — are shipping with the current reclassification cycle, and records gain them as domains are re-verified. The original twenty types are present throughout. Deliveries and API responses always reflect the current state of each record, and monthly refresh subscribers receive the action types automatically as re-verification progresses.
No. The database is built only from pages a site links from its own homepage structure — navigation, footers, sitemaps and internal cross-references — never by probing for unadvertised admin or login paths, and never by submitting forms. URL-level coverage of admin and write endpoints comes from the rules file and the curated host list instead.
The full repository exceeds 40 million domains, selected from real-world browsing popularity, and represents 99.99% of active internet usage — traffic is concentrated enough that the remaining long tail of parked and dormant domains is rarely encountered by an agent. Licensed tiers of 10M, 15M and 30M domains are cut from the repository by popularity rank; custom slices are available on request.
CSV (one row per domain, the default), JSON / JSON Lines (nested documents, matching what the lookup API returns), or an importable SQL dump with indexed domain lookups. All formats carry the same schema: page-type URLs, IAB categories, filtering category, popularity ranks, country, language, personas, OpenPageRank and classification metadata.
A one-time purchase includes no updates: your copy is frozen at purchase-date accuracy while the web moves — page-type URLs change as sites restructure, expired domains get re-registered by new owners with different content, and new domains appear that your copy has never seen. The monthly refresh option delivers a re-verified database every month, including roughly 300,000 newly registered domains screened per update cycle. For production guardrails we recommend the refresh option.
API access starts at $99/month. Full database licenses are $14,999 for 10M domains, $24,999 for 15M domains and $49,999 for 30M domains, each available one-time or with monthly refreshes. OEM and custom licensing is priced individually — see the pricing page or contact us.
Yes — the free 100-domain sample uses the exact schema and format of a licensed delivery, with full records for well-known domains, and requires no signup. Download it, verify the URLs against the live sites, and load it into your pipeline as a dry run.
Alpha Quantum, a Munich-based data company. The IAB classifier and the 59-category filtering taxonomy in this product are the same systems behind our website-categorization and web-filtering databases, which are licensed by software and network-security vendors. The page-type layer was built specifically for agent access control.
100 well-known domains, full schema, real URLs — free, no signup. Then pick a tier on the pricing page.
