How common are login, pricing, documentation and careers pages across the web? Aggregate counts from a page-type database of 40,786,727 domains, the data behind page-level egress control for AI agents. Free under CC BY 4.0.
Share of the 16,448,576 domains with at least one verified page type. Login, the page type an agent should never submit to, is highlighted.
The page-type database behind the AI agent allow list records, for each domain, the verified URLs of its login, signup, pricing, documentation and other pages. An agent policy can then allow reads and deny writes at page level instead of trusting or blocking a whole site. This release publishes the aggregate picture: how often each page type occurs, how many page types a domain typically has, how that varies across 40 site languages and 60 top-level domains, which page types appear together, where the pages live (subdomains, query strings, path depth), and the categories of the separate host list. It also runs the 40 URL rules of the allow list on every verified URL: a rule matches 0.21% of read pages such as documentation or pricing, most of them registration or login pages filed under a read type (about 0.05% are genuine false positives), and 46% of login pages. The database itself stays commercial; besides the aggregates, the release includes the 100-domain sample with verified URLs.
Use it to estimate how much of the web exposes login or upload pages to autonomous agents, to size page-level policies, or as a reference distribution for web-measurement research.
page-type-prevalence.csvDomains with a verified URL for each of the 20 page types, 1 KBpage-types-per-domain.csvHow many page types each domain has, 1 KBlanguages.csvDomains per site language (40 languages), 1 KBlanguage-page-types.csvPage-type prevalence for all 40 languages, 25 KBpage-type-cooccurrence.csvHow often two page types occur on the same domain, with conditional shares and lift, 10 KBtlds.csvThe 60 largest top-level domains, with the share that has page types, 2 KBtld-page-types.csvPage-type prevalence per top-level domain, 38 KBurl-structure-by-page-type.csvWhere each page type lives: HTTPS, subdomains, query strings, path depth, 2 KBrule-layer-by-page-type.csvThe 40 URL rules run on every verified URL: match rate per page type, 2 KBrule-layer-first-rule.csvWhich rule matched first, per page type, 12 KBsample-100-domains.csv100 domains with their verified URLs, categories and popularity group, 95 KBhost-list-categories.csvHost-list categories with default action and number of hosts, 1 KBREADME.mdDataset description, 6 KBDATA_DICTIONARY.mdColumns explained, 4 KBLICENSE.txtCC BY 4.0 licence, 1 KBCITATION.txtHow to cite, 1 KBCHANGELOG.mdVersion history, 1 KBmanifest.jsonFile sizes and checksums (JSON), 3 KBSHA256SUMSChecksums, 2 KBVersion 1.1.0, database snapshot of September 2026. DOI 10.6084/m9.figshare.34028775.
Limitations: counts describe the database at one point in time, not the live web; a missing page type means no verified URL was found; labels come from an automatic classifier. Related open data: the AI tools taxonomy dataset and the web filtering research dataset.