# AI Agent Page-Type Statistics

Version 1.1.0. September 2026.

Aggregate statistics from a page-type database that covers 40,786,727 domains. For each domain, the database records which kinds of page it has, such as login, pricing, documentation or careers pages, and their verified URLs. AI agents that browse the web act on exactly these pages: they read documentation and pricing pages, but they also reach login, signup, upload and checkout pages. Knowing which page types exist, and how common they are, is the basis for [page-level egress control for AI agents](https://www.aiagentallowlist.com/page-types-database.php).

It is published by AI Agent Allow List, a [zero trust AI agent](https://www.aiagentallowlist.com/) policy service. The database itself is commercial. This release contains aggregate counts and shares over all 40,786,727 domains, plus a sample of 100 domains with their verified URLs (`sample-100-domains.csv`), so the record format can be inspected.

## At a glance

| Measure | Value |
|---|---:|
| Domains covered | 40,786,727 |
| Domains with at least one verified page type | 16,448,576 (40.3%) |
| Page types | 20 |
| Mean page types per domain (domains with at least one) | 3.61 |
| Most common page type | contact (70.4% of domains with page types) |
| Domains with a verified login page | 3,257,119 (19.8% of domains with page types) |
| Site languages covered | 40 |
| Top-level domains covered in detail | 60 |
| Verified URLs checked against the 40-rule layer | 58,814,640 |
| Read-page URLs matched by a rule | 115,837 of 55,565,543 (0.21%) |
| Login URLs matched by a rule | 45.8% |

## Files

| File | Content |
|---|---|
| `page-type-prevalence.csv` | 20 rows: domains with a verified URL for each page type, and shares |
| `page-types-per-domain.csv` | How many page types each domain has (0 to 19) |
| `languages.csv` | 40 rows: domains per site language |
| `language-page-types.csv` | Page-type prevalence for all 40 languages (800 rows) |
| `page-type-cooccurrence.csv` | How often two page types occur on the same domain, with conditional shares and lift |
| `tlds.csv` | The 60 largest top-level domains: domains and share with page types |
| `tld-page-types.csv` | Page-type prevalence per top-level domain (1,200 rows) |
| `url-structure-by-page-type.csv` | Structure of the verified URLs per page type: HTTPS, same host / subdomain / other host, query strings, file extensions, path depth |
| `rule-layer-by-page-type.csv` | The URL rule layer run on every verified URL: how often a rule matches each page type |
| `rule-layer-first-rule.csv` | Which rule matched first, per page type (426 rows) |
| `sample-100-domains.csv` | 100 domains with verified URLs per page type, language, IAB categories, web-filtering category, country and popularity group |
| `host-list-categories.csv` | The categories of the separate host list (cloud consoles, metadata endpoints, package registries, paste sites...) with default action and number of hosts |
| `DATA_DICTIONARY.md` | Every column explained |
| `CITATION.txt`, `LICENSE.txt`, `CHANGELOG.md` | Citation, licence and version history |
| `manifest.json`, `SHA256SUMS` | File sizes and checksums |

## Method

Page types were identified by a machine-learning pipeline that assigns URLs on each domain to a fixed set of 20 page types; quality was checked on a manually labelled sample. A page type counts for a domain when at least one verified URL of that type was found. Language is the detected main language of the site. The commercial database also tracks further page types, for example signup, cart, checkout and upload pages; those are not part of the aggregate tables (the 100-domain sample shows them where present).

The rule layer is the set of 40 URL rules used for page-level egress control of AI agents (login, signup, checkout, upload, admin panels, repository writes and similar). Each verified URL was tested against all rules; the tables report whether any rule matched and which rule matched first (30 of the 40 rules matched first at least once). On the full database, 0.21% of read-page URLs are matched by a rule, in line with the 0.22% measured on a 1% sample in the accompanying paper. A match on a read page (documentation, pricing, blog and so on) stops an agent on that page. It is not always an error: in a manual review of a random sample of such matches, about four in five were registration, login, cart or administration pages that the page-type classifier had filed under a read type (for example a `/register` link labelled as pricing), which is why pricing has the highest match rate. The genuine false-positive rate is about 0.05% of read pages. Rule set as of 30 September 2026. For login pages a match is the intended outcome; login URLs that match no rule are still covered by the default for writes, since submitting a login form is a POST. The 100-domain sample is the one already published at https://www.aiagentallowlist.com/page-types-database.php, unchanged.

## Who this is for

- **Agent-security and AI-safety researchers** designing egress policies for autonomous agents.
- **Web-measurement researchers** studying how sites are structured across languages.
- **Security teams** estimating how much of the web exposes login, signup or upload pages to agents.

## Limitations

- Counts describe the database at one point in time (September 2026), not the live web.
- A missing page type means no verified URL was found, not that the site certainly lacks the page.
- Page-type labels come from an automatic classifier and contain errors.
- URL structure and rule results use one verified URL per domain and page type, not every URL of the site.
- The 100-domain sample is not random: it was chosen to show well-known sites with many page types.

## Citation

Alpha Quantum (2026). *AI Agent Page-Type Statistics* (Version 1.1.0). AI Agent Allow List. https://www.aiagentallowlist.com/ai-agent-page-types-dataset.php

## Licence

Creative Commons Attribution 4.0 International (CC BY 4.0). See `LICENSE.txt`.
