How common are login, pricing, documentation and careers pages across the web? Aggregate counts from a page-type database of 40,786,727 domains, the data behind page-level egress control for AI agents. Free under CC BY 4.0.
Share of the 16,448,576 domains with at least one verified page type. Login, the page type an agent should never submit to, is highlighted.
The page-type database behind the AI agent allow list records, for each domain, the verified URLs of its login, signup, pricing, documentation and other pages. An agent policy can then allow reads and deny writes at page level instead of trusting or blocking a whole site. This release publishes the aggregate picture: how often each page type occurs, how many page types a domain typically has, how that varies across 40 site languages, which page types appear together, and the categories of the separate host list. It contains no domains and no URLs.
Use it to estimate how much of the web exposes login or upload pages to autonomous agents, to size page-level policies, or as a reference distribution for web-measurement research.
page-type-prevalence.csvDomains with a verified URL for each of the 20 page types, 1 KBpage-types-per-domain.csvHow many page types each domain has, 1 KBlanguages.csvDomains per site language (40 languages), 1 KBlanguage-page-types.csvPage-type prevalence for the largest languages, 6 KBpage-type-cooccurrence.csvHow often two page types occur on the same domain, 6 KBhost-list-categories.csvHost-list categories with default action and number of hosts, 1 KBREADME.mdDataset description, 3 KBDATA_DICTIONARY.mdColumns explained, 1 KBLICENSE.txtCC BY 4.0 licence, 1 KBCITATION.txtHow to cite, 1 KBCHANGELOG.mdVersion history, 1 KBmanifest.jsonFile sizes and checksums (JSON), 2 KBSHA256SUMSChecksums, 1 KBVersion 1.0.0, database snapshot of September 2026.
Limitations: counts describe the database at one point in time, not the live web; a missing page type means no verified URL was found; labels come from an automatic classifier. Related open data: the AI tools taxonomy dataset and the web filtering research dataset.
