This is the field-by-field reference for the AI Agent Allowlist database: what each column contains, what it is derived from, and what an absent value means. Every other page on this site that mentions a database field links back here.
This schema is what makes prevention verifiable, not just claimed. Several 2026 agent incidents map cleanly onto specific fields below — the login and checkout keys in page_types most directly. Would your agents have been stopped? check the incident analysis.
The database is a flat table: domain is the primary key, and every other field describes something true about that domain as of the last classification pass. There is no nested schema to parse beyond the page_types field itself, which carries up to 28 verified URLs.
Two design choices matter for anyone integrating against this schema. First, absence is recorded, not omitted: if a domain has no verified checkout URL, the schema says so explicitly rather than leaving a lookup to guess whether the field was never checked or genuinely does not exist. Second, every URL in page_types is verified from the domain's live link structure at classification time, never a guessed path pattern like assuming /login exists because most sites use it.
This table lists only fields actually present in the database and the free sample CSV. Column order matches the CSV header exactly.
| # | Field | Type | Description |
|---|---|---|---|
| 1 | domain primary key | string | The registrable domain, e.g. stripe.com. One row per domain in the database. |
| 2 | page_types | map | Up to 28 type=URL pairs (login, signup, checkout, pricing, documentation and 24 more), each URL verified from the domain's live link structure. Types with no verified URL are recorded as absent, never guessed. |
| 3 | language | string | Primary content language detected for the domain. |
| 4 | IAB v2 Tier 1 Category | string | Top-level IAB v2 content category, from the 700+ category taxonomy. |
| 5 | IAB v2 Tier 2 Category | string | Second-level IAB v2 category, nested under Tier 1. |
| 6 | IAB v2 Tier 3 Category | string | Third-level IAB v2 category, present where the taxonomy defines one. |
| 7 | IAB v2 Tier 4 Category | string | Fourth-level IAB v2 category, the most granular v2 classification available. |
| 8 | IAB v3 Tier 1 Category | string | Top-level IAB v3 content category. Run in parallel with v2 since buyers standardize on different IAB versions. |
| 9 | IAB v3 Tier 2 Category | string | Second-level IAB v3 category. |
| 10 | IAB v3 Tier 3 Category | string | Third-level IAB v3 category. |
| 11 | IAB v3 Tier 4 Category | string | Fourth-level IAB v3 category, most granular v3 classification available. |
| 12 | Web Filtering Category | string | One category from a 59-category filtering taxonomy, separate from IAB and tuned for access-control decisions rather than ad targeting. |
| 13 | Personas | list | Audience personas describing who the domain serves, drawn from a set of 2,000+ defined personas. |
| 14 | OpenPageRank | float | Link-authority score for the domain, usable for trust-weighting or discounting low-authority/spam domains. |
| 15 | Country | string | Country associated with the domain. |
| 16 | Global Popularity Rank Group | string | Banded global traffic-popularity group for the domain (a group, not an exact numeric rank). |
| 17 | Country Level Popularity Rank Group | string | Banded popularity group for the domain within its own country. |
| 18 | num_distinct_page_types | int | Count of how many of the 28 page types were actually found and verified for this domain, from 0 up to 28. |
| 19 | classification metadata | date | Last-processed date recording when the domain's row was most recently classified or re-verified. |
Rows 1-17 match the free sample CSV column-for-column; classification metadata ships with licensed and API deliveries. See coverage tiers for how domain volume maps to license price, and API docs for how these fields appear in a JSON lookup response.
Field 2 in the table above, page_types, is itself a map of up to 28 keys. This is the full enumeration, grouped the way the underlying classification pipeline groups them.
| Page type key | Group | Kind |
|---|---|---|
| login | Identity | Read |
| signup | Identity | action type |
| password_reset | Identity | action type |
| cart | Commerce | action type |
| checkout | Commerce | action type |
| subscribe | Commerce | action type |
| pricing | Commerce | Read |
| product | Commerce | Read |
| post_create | Content-write | action type |
| comment | Content-write | action type |
| upload | Content-write | action type |
| community | Content-write | Read |
| documentation | Research/read | Read |
| help_center | Research/read | Read |
| integrations | Research/read | Read |
| status | Research/read | Read |
| security | Research/read | Read |
| contact | Research/read | Read |
| about | Research/read | Read |
| leadership | Research/read | Read |
| careers | Research/read | Read |
| blog | Research/read | Read |
| press | Research/read | Read |
| events | Research/read | Read |
| partners | Research/read | Read |
| case_studies | Research/read | Read |
| legal | Research/read | Read |
| sitemap | Research/read | Read |
The 8 fields tagged action type ship with the current reclassification cycle. Not every domain has verified URLs for all 8 yet; where one is missing, page_types records its absence rather than omitting the key. A full narrative walkthrough of each group lives on the page-types database overview.
Registrable domain only, not full hostname. Subdomain-specific behavior, such as an identity provider hosted on a separate subdomain, is captured inside the relevant page_type URL, not as a separate row.
A verified URL means the classification pipeline found and confirmed that link in the domain's live link structure, built from analysis of 10 billion+ links. It is never a pattern guess like assuming /login exists.
Both versions are carried because buyers standardize on different IAB releases. Tiers 3 and 4 are populated where the taxonomy defines a category at that depth; not every domain resolves to four levels.
Deliberately separate from IAB. IAB is built for advertising and content classification; the 59-category filtering taxonomy is built specifically for access-control and policy decisions.
Personas describe who a domain serves (for example, "enterprise IT buyer" or "K-12 educator"), drawn from a defined set of 2,000+, and support agent policies that vary by target audience rather than by category alone.
A link-authority score, not a popularity rank. Useful for down-weighting low-authority domains independent of how much raw traffic they get.
Both are banded groups, not exact numeric ranks, because exact rank volatility would make the field noisy at the tail; the group is stable enough to build policy tiers on.
A quick coverage signal per domain: a high number means the domain is richly classified across many page types, useful for sorting or filtering a bulk export by classification depth.
Carries the last-processed date so a consumer can decide how fresh a given row is relative to their own refresh cadence, independent of the licensing refresh cycle.
These figures describe the schema's structure, not a claim about any specific domain's data quality.
Three design decisions run through every field above, and understanding them explains why the schema looks the way it does rather than, say, a deeply nested document per domain.
The first decision is flatness. Every consumer of this data, whether a proxy's policy engine, a data warehouse, or a spreadsheet, needs to join on domain quickly and predictably. A flat row per domain, with page_types as the one structured field, keeps that join trivial regardless of the target system: CSV, JSON Lines, or an indexed SQL table all represent this schema without translation loss. Full delivery format options are covered on the page-types database page.
The second is verification over inference. Every URL that appears anywhere in page_types was found in the domain's actual link structure during classification, not synthesized from a common path convention. This matters specifically for the highest-stakes page types — login above all — where guessing produces confident-looking wrong answers far more often than an honest "not verified" would. The mechanics of exactly how this works for login pages specifically, including subdomain-hosted identity providers and locale-prefixed paths, are documented on login page detection.
The third is explicit absence. A schema that simply omits a key when no URL was found is indistinguishable, to a downstream consumer, from a schema that never checked for that key at all. Recording absence explicitly means a policy engine can treat "no checkout page found on this domain" as a known fact rather than an unknown one, which matters when the policy decision is "deny by default for anything not resolved."
The schema is not static, and it is worth documenting the one structural change that matters for anyone who integrated against an earlier version.
The original classification pipeline covered 20 page types, all of them read-oriented: documentation, pricing, login, contact, careers and the rest of the research and identity surfaces a browsing agent needs to navigate to safely. The current reclassification cycle added 8 additional keys, all of them action types: signup, password_reset, cart, checkout, upload, post_create, comment and subscribe. The distinction between the original 20 and the newer 8 is meaningful, not cosmetic: action types describe pages where an agent would do something — create an account, submit a payment, publish content — rather than simply read something, which is exactly the category of page most agent-safety policies need to deny by default.
Because the 8 action types shipped more recently, coverage across the 40M-domain repository for them is still filling in; not every domain has a verified signup or checkout URL yet, even domains that are otherwise richly classified across the original 20 read types. The num_distinct_page_types field, described above, is the fastest way to see how deep a given domain's classification actually goes, including whether it has been reached by the action-type reclassification pass yet.
The 19 fields above do not change across delivery formats; only their on-the-wire representation does.
A CSV export, like the free sample, represents page_types as a single delimited column, typically type=url pairs separated by a consistent delimiter, because CSV has no native concept of nested structure. A JSON or JSON Lines export represents the same field as a proper nested object, one key per page type, which is also the shape the lookup API returns for a single-URL check. An indexed SQL dump, the third option offered with a full database license, typically normalizes page_types into a second table keyed by domain and page type, so a query can filter for "every domain with a verified checkout URL" without parsing a delimited string first. Whichever format a given integration uses, the underlying 19-field schema documented on this page is the same; only the packaging differs.
The full schema above is what a licensed on-prem database delivers per row. The self-serve lookup API returns a narrower, decision-focused slice of the same underlying classification for a single URL at a time.
A licensed database consumer typically loads the full 19-field schema into their own data warehouse or policy store and queries it however their infrastructure prefers. A lookup API consumer instead sends one URL and receives a decision plus the matched page type, for example a request to /api/check?url=https://stripe.com/login returning "result":"deny","id":"login". Both paths draw from the identical classification work; they differ only in whether the schema is delivered wholesale or resolved per request. Rate limits, authentication and full response shapes for the API path are documented at the API docs.
Which path fits depends mostly on call volume and latency tolerance, not on which fields are available. A team making a small number of navigation decisions per agent run, or evaluating the product before committing, is usually better served by the lookup API's per-URL simplicity. A team running a high-volume gateway checking every outbound request from a large agent fleet typically outgrows the API's fair-use ceiling on live lookups and moves to an on-prem license precisely to remove the network round-trip and the per-call cost, running the same schema as a local index instead.
See how domain volume (10M/15M/30M/40M+) maps to license price and to real-world navigation coverage on coverage tiers.
A deep technical reference on how the login key specifically is found and verified, and the edge cases that make guessing unreliable, on login page detection.
Why a hand-maintained list cannot replicate this schema's coverage at any team size, worked through in detail, on manual curation vs. the database.
For a coarser, domain-only categorization dataset without page-type granularity, see the sibling product Web Filtering Database, which shares the same Web Filtering Category taxonomy referenced in field 12 above. For how these fields combine into an actual enforcement policy rather than a static export, see agent guardrails.
A schema is easy to admire in the abstract. It is worth grounding this one in a specific, documented case where a single field made the difference.
In the 2026 account-takeover campaign analyzed on this site, agents from the same operation reached four third-party accounts by requesting login pages, and in some cases signup or password-reset flows, on services they did not control. Our analysis shows that this is precisely the login, signup and password_reset triad inside the page_types field, all three tagged identity-group fields in the table above. Denying those three keys by default, rather than trying to infer intent from what the agent claimed it was trying to do, is what our analysis shows would have stopped every documented authentication hop in that campaign before it happened. The full incident walkthrough, including why login coverage is deliberately checked twice, by both the verified database key and a pattern-matching egress rule, is on the account-takeovers incident analysis.
In the delivered schema it is one structured field, a map of up to 28 type=URL pairs. The free sample CSV represents it as a single page_types column; JSON and JSON Lines deliveries represent it as a nested object with one key per verified page type.
The absence is recorded explicitly rather than the key being silently omitted, so a consumer can distinguish "checked and not found" from "not yet checked."
Different buyers and downstream systems standardize on different IAB taxonomy versions. Carrying both avoids forcing a migration on either group and lets a single database serve both integration paths.
Personas are drawn from a defined set of more than 2,000 audience personas, assigned per domain based on who the site's content and structure indicate it serves.
No. It counts only the page types with an actual verified URL for that domain, from 0 to 28, so it functions as a classification-depth signal.
The free 100-domain sample CSV at /sample_databases/agent_allowlist_sample.csv uses this exact column order and field set on real, well-known domains.
Download the free 100-domain sample or review database license pricing.