# Data dictionary

All counts refer to the page-type database snapshot of September 2026 (40,786,727 domains; 16,448,576 with at least one verified page type).

## page-type-prevalence.csv
- `page_type`: one of contact, about, legal, product, blog, login, help_center, pricing, events, careers, case_studies, leadership, sitemap, partners, press, documentation, security, status, community, integrations.
- `domains`: domains with at least one verified URL of this type.
- `share_of_domains_with_page_types`: `domains` / 16,448,576.
- `share_of_all_domains`: `domains` / 40,786,727.

## page-types-per-domain.csv
- `page_types_per_domain`: number of distinct page types found on a domain (0 = none found).
- `domains`, `share_of_all_domains`.

## languages.csv
- `language_code`: ISO 639-1 code of the site's detected main language; `language`: English name.
- `domains`, `share_of_all_domains`.

## language-page-types.csv
- `language_code`, `page_type`, `domains`.
- `share_of_language_domains`: `domains` / all domains of that language (including domains without page types).
- `share_of_language_domains_with_page_types`: `domains` / domains of that language with at least one page type.

## page-type-cooccurrence.csv
- `page_type_a`, `page_type_b`: an unordered pair of page types.
- `domains_with_both`: domains with verified URLs of both types; `share_of_domains_with_page_types`: / 16,448,576.
- `share_of_a_with_b`: of the domains with type a, the share that also has type b; `share_of_b_with_a` the reverse.
- `lift`: observed / expected co-occurrence if the two types were independent (above 1 = found together more often than chance).

## tlds.csv
- `tld`: last label of the domain (e.g. `com`, `de`). `domains`, `share_of_all_domains`.
- `domains_with_page_types`, `share_with_page_types`: domains of that TLD with at least one verified page type.

## tld-page-types.csv
- `tld`, `page_type`, `domains`; `share_of_tld_domains` and `share_of_tld_domains_with_page_types`.

## url-structure-by-page-type.csv
- `urls`: verified URLs of this type (one per domain).
- `share_https`: URL uses HTTPS. `share_same_host`: URL is on the domain itself or its www host; `share_subdomain`: on another subdomain (e.g. `help.`, `docs.`, `account.`); `share_other_host`: on a different domain (hosted help desks, careers portals, identity providers).
- `share_with_query`: URL has a query string. `share_file_extension`: path ends in .php, .asp(x), .jsp, .cgi or .htm(l).
- `share_depth_0` ... `share_depth_5_plus`: number of path segments (0 = site root).

## rule-layer-by-page-type.csv
- `urls`: verified URLs of this type. `urls_matched_by_a_rule`, `share_matched`: URLs where at least one of the URL rules matched.
- `interpretation`: for read pages a match blocks the page; most such matches are mislabelled identity or commerce URLs, the rest are false positives. For login pages a match is the intended outcome.

## rule-layer-first-rule.csv
- `page_type`, `rule_id` (the first rule that matched, e.g. `login`, `signup`, `cart`, `upload`), `urls`, `share_of_page_type_urls`.

## sample-100-domains.csv
- `domain`, `language`, `num_distinct_page_types`.
- `page_types`: `type=URL` pairs separated by `;`. Includes further page types of the commercial database (e.g. signup, subscribe) where present.
- IAB Content Taxonomy v2 and v3 categories (tiers 1 to 4), `Web Filtering Category`, `Personas`, `OpenPageRank`, `Country`, `Global Popularity Rank Group`, `Country Level Popularity Rank Group`.
- Columns: `domain`, `page_types`, `language`, `IAB v2 Tier 1 Category`, `IAB v2 Tier 2 Category`, `IAB v2 Tier 3 Category`, `IAB v2 Tier 4 Category`, `IAB v3 Tier 1 Category`, `IAB v3 Tier 2 Category`, `IAB v3 Tier 3 Category`, `IAB v3 Tier 4 Category`, `Web Filtering Category`, `Personas`, `OpenPageRank`, `Country`, `Global Popularity Rank Group`, `Country Level Popularity Rank Group`, `num_distinct_page_types`.

## host-list-categories.csv
- `host_category`: category of the separate host list (hosts that are denied or flagged regardless of page type).
- `default_action`: `deny` or `flag`. `hosts`: number of hosts in the category.
