AI Agent Allowlist
Home Page-Types Database Agent Guardrails 2026 Incidents API Docs Pricing
Resources
Use Cases Industries & Buyers Learn: Core Concepts Implementation Guides Comparisons Schema & Data Reference FAQ Glossary
Why It Matters
2026 Agent Incidents Category Targeting Database Refreshes Contact Customer Login
Download Free Sample
the authoritative schema

Every column in the database. Documented once, exactly.

This is the field-by-field reference for the AI Agent Allowlist database: what each column contains, what it is derived from, and what an absent value means. Every other page on this site that mentions a database field links back here.

0Page-type fields
0Top-level schema columns
0Combined IAB v2 + v3 tier columns
0Domains carrying this schema

This schema is what makes prevention verifiable, not just claimed. Several 2026 agent incidents map cleanly onto specific fields below — the login and checkout keys in page_types most directly. Would your agents have been stopped? check the incident analysis.

Design principle

One row per domain, one column per fact

The database is a flat table: domain is the primary key, and every other field describes something true about that domain as of the last classification pass. There is no nested schema to parse beyond the page_types field itself, which carries up to 28 verified URLs.

Two design choices matter for anyone integrating against this schema. First, absence is recorded, not omitted: if a domain has no verified checkout URL, the schema says so explicitly rather than leaving a lookup to guess whether the field was never checked or genuinely does not exist. Second, every URL in page_types is verified from the domain's live link structure at classification time, never a guessed path pattern like assuming /login exists because most sites use it.

Master schema

The 17 top-level columns, field by field

This table lists only fields actually present in the database and the free sample CSV. Column order matches the CSV header exactly.

#FieldTypeDescription
1domain primary keystringThe registrable domain, e.g. stripe.com. One row per domain in the database.
2page_typesmapUp to 28 type=URL pairs (login, signup, checkout, pricing, documentation and 24 more), each URL verified from the domain's live link structure. Types with no verified URL are recorded as absent, never guessed.
3languagestringPrimary content language detected for the domain.
4IAB v2 Tier 1 CategorystringTop-level IAB v2 content category, from the 700+ category taxonomy.
5IAB v2 Tier 2 CategorystringSecond-level IAB v2 category, nested under Tier 1.
6IAB v2 Tier 3 CategorystringThird-level IAB v2 category, present where the taxonomy defines one.
7IAB v2 Tier 4 CategorystringFourth-level IAB v2 category, the most granular v2 classification available.
8IAB v3 Tier 1 CategorystringTop-level IAB v3 content category. Run in parallel with v2 since buyers standardize on different IAB versions.
9IAB v3 Tier 2 CategorystringSecond-level IAB v3 category.
10IAB v3 Tier 3 CategorystringThird-level IAB v3 category.
11IAB v3 Tier 4 CategorystringFourth-level IAB v3 category, most granular v3 classification available.
12Web Filtering CategorystringOne category from a 59-category filtering taxonomy, separate from IAB and tuned for access-control decisions rather than ad targeting.
13PersonaslistAudience personas describing who the domain serves, drawn from a set of 2,000+ defined personas.
14OpenPageRankfloatLink-authority score for the domain, usable for trust-weighting or discounting low-authority/spam domains.
15CountrystringCountry associated with the domain.
16Global Popularity Rank GroupstringBanded global traffic-popularity group for the domain (a group, not an exact numeric rank).
17Country Level Popularity Rank GroupstringBanded popularity group for the domain within its own country.
18num_distinct_page_typesintCount of how many of the 28 page types were actually found and verified for this domain, from 0 up to 28.
19classification metadatadateLast-processed date recording when the domain's row was most recently classified or re-verified.

Rows 1-17 match the free sample CSV column-for-column; classification metadata ships with licensed and API deliveries. See coverage tiers for how domain volume maps to license price, and API docs for how these fields appear in a JSON lookup response.

Inside page_types

The 28 page types, grouped and flagged

Field 2 in the table above, page_types, is itself a map of up to 28 keys. This is the full enumeration, grouped the way the underlying classification pipeline groups them.

Page type keyGroupKind
loginIdentityRead
signupIdentityaction type
password_resetIdentityaction type
cartCommerceaction type
checkoutCommerceaction type
subscribeCommerceaction type
pricingCommerceRead
productCommerceRead
post_createContent-writeaction type
commentContent-writeaction type
uploadContent-writeaction type
communityContent-writeRead
documentationResearch/readRead
help_centerResearch/readRead
integrationsResearch/readRead
statusResearch/readRead
securityResearch/readRead
contactResearch/readRead
aboutResearch/readRead
leadershipResearch/readRead
careersResearch/readRead
blogResearch/readRead
pressResearch/readRead
eventsResearch/readRead
partnersResearch/readRead
case_studiesResearch/readRead
legalResearch/readRead
sitemapResearch/readRead

The 8 fields tagged action type ship with the current reclassification cycle. Not every domain has verified URLs for all 8 yet; where one is missing, page_types records its absence rather than omitting the key. A full narrative walkthrough of each group lives on the page-types database overview.

Field notes

Details worth knowing before you integrate

domain

Registrable domain only, not full hostname. Subdomain-specific behavior, such as an identity provider hosted on a separate subdomain, is captured inside the relevant page_type URL, not as a separate row.

page_types

A verified URL means the classification pipeline found and confirmed that link in the domain's live link structure, built from analysis of 10 billion+ links. It is never a pattern guess like assuming /login exists.

IAB v2 vs. v3

Both versions are carried because buyers standardize on different IAB releases. Tiers 3 and 4 are populated where the taxonomy defines a category at that depth; not every domain resolves to four levels.

Web Filtering Category

Deliberately separate from IAB. IAB is built for advertising and content classification; the 59-category filtering taxonomy is built specifically for access-control and policy decisions.

Personas

Personas describe who a domain serves (for example, "enterprise IT buyer" or "K-12 educator"), drawn from a defined set of 2,000+, and support agent policies that vary by target audience rather than by category alone.

OpenPageRank

A link-authority score, not a popularity rank. Useful for down-weighting low-authority domains independent of how much raw traffic they get.

Popularity Rank Groups

Both are banded groups, not exact numeric ranks, because exact rank volatility would make the field noisy at the tail; the group is stable enough to build policy tiers on.

num_distinct_page_types

A quick coverage signal per domain: a high number means the domain is richly classified across many page types, useful for sorting or filtering a bulk export by classification depth.

classification metadata

Carries the last-processed date so a consumer can decide how fresh a given row is relative to their own refresh cadence, independent of the licensing refresh cycle.

Schema at a glance

Coverage of the schema across the sample

These figures describe the schema's structure, not a claim about any specific domain's data quality.

19Total documented fields
28Keys inside page_types
8IAB v2+v3 tier columns
59Web filtering categories
Why the schema is this shape

Flat rows, verified URLs, explicit absence

Three design decisions run through every field above, and understanding them explains why the schema looks the way it does rather than, say, a deeply nested document per domain.

The first decision is flatness. Every consumer of this data, whether a proxy's policy engine, a data warehouse, or a spreadsheet, needs to join on domain quickly and predictably. A flat row per domain, with page_types as the one structured field, keeps that join trivial regardless of the target system: CSV, JSON Lines, or an indexed SQL table all represent this schema without translation loss. Full delivery format options are covered on the page-types database page.

The second is verification over inference. Every URL that appears anywhere in page_types was found in the domain's actual link structure during classification, not synthesized from a common path convention. This matters specifically for the highest-stakes page types — login above all — where guessing produces confident-looking wrong answers far more often than an honest "not verified" would. The mechanics of exactly how this works for login pages specifically, including subdomain-hosted identity providers and locale-prefixed paths, are documented on login page detection.

The third is explicit absence. A schema that simply omits a key when no URL was found is indistinguishable, to a downstream consumer, from a schema that never checked for that key at all. Recording absence explicitly means a policy engine can treat "no checkout page found on this domain" as a known fact rather than an unknown one, which matters when the policy decision is "deny by default for anything not resolved."

Schema history

Why the page_types field grew from 20 keys to 28

The schema is not static, and it is worth documenting the one structural change that matters for anyone who integrated against an earlier version.

The original classification pipeline covered 20 page types, all of them read-oriented: documentation, pricing, login, contact, careers and the rest of the research and identity surfaces a browsing agent needs to navigate to safely. The current reclassification cycle added 8 additional keys, all of them action types: signup, password_reset, cart, checkout, upload, post_create, comment and subscribe. The distinction between the original 20 and the newer 8 is meaningful, not cosmetic: action types describe pages where an agent would do something — create an account, submit a payment, publish content — rather than simply read something, which is exactly the category of page most agent-safety policies need to deny by default.

Because the 8 action types shipped more recently, coverage across the 40M-domain repository for them is still filling in; not every domain has a verified signup or checkout URL yet, even domains that are otherwise richly classified across the original 20 read types. The num_distinct_page_types field, described above, is the fastest way to see how deep a given domain's classification actually goes, including whether it has been reached by the action-type reclassification pass yet.

Delivery formats

The same schema, three different shapes on the wire

The 19 fields above do not change across delivery formats; only their on-the-wire representation does.

A CSV export, like the free sample, represents page_types as a single delimited column, typically type=url pairs separated by a consistent delimiter, because CSV has no native concept of nested structure. A JSON or JSON Lines export represents the same field as a proper nested object, one key per page type, which is also the shape the lookup API returns for a single-URL check. An indexed SQL dump, the third option offered with a full database license, typically normalizes page_types into a second table keyed by domain and page type, so a query can filter for "every domain with a verified checkout URL" without parsing a delimited string first. Whichever format a given integration uses, the underlying 19-field schema documented on this page is the same; only the packaging differs.

How this schema is used downstream

From a database row to a single API answer

The full schema above is what a licensed on-prem database delivers per row. The self-serve lookup API returns a narrower, decision-focused slice of the same underlying classification for a single URL at a time.

A licensed database consumer typically loads the full 19-field schema into their own data warehouse or policy store and queries it however their infrastructure prefers. A lookup API consumer instead sends one URL and receives a decision plus the matched page type, for example a request to /api/check?url=https://stripe.com/login returning "result":"deny","id":"login". Both paths draw from the identical classification work; they differ only in whether the schema is delivered wholesale or resolved per request. Rate limits, authentication and full response shapes for the API path are documented at the API docs.

Which path fits depends mostly on call volume and latency tolerance, not on which fields are available. A team making a small number of navigation decisions per agent run, or evaluating the product before committing, is usually better served by the lookup API's per-URL simplicity. A team running a high-volume gateway checking every outbound request from a large agent fleet typically outgrows the API's fair-use ceiling on live lookups and moves to an on-prem license precisely to remove the network round-trip and the per-call cost, running the same schema as a local index instead.

Related reference pages

Where to go next

Coverage

See how domain volume (10M/15M/30M/40M+) maps to license price and to real-world navigation coverage on coverage tiers.

Login detail

A deep technical reference on how the login key specifically is found and verified, and the edge cases that make guessing unreliable, on login page detection.

Curation math

Why a hand-maintained list cannot replicate this schema's coverage at any team size, worked through in detail, on manual curation vs. the database.

For a coarser, domain-only categorization dataset without page-type granularity, see the sibling product Web Filtering Database, which shares the same Web Filtering Category taxonomy referenced in field 12 above. For how these fields combine into an actual enforcement policy rather than a static export, see agent guardrails.

The schema, proven against a real incident

What field 2 actually stopped in 2026

A schema is easy to admire in the abstract. It is worth grounding this one in a specific, documented case where a single field made the difference.

In the 2026 account-takeover campaign analyzed on this site, agents from the same operation reached four third-party accounts by requesting login pages, and in some cases signup or password-reset flows, on services they did not control. Our analysis shows that this is precisely the login, signup and password_reset triad inside the page_types field, all three tagged identity-group fields in the table above. Denying those three keys by default, rather than trying to infer intent from what the agent claimed it was trying to do, is what our analysis shows would have stopped every documented authentication hop in that campaign before it happened. The full incident walkthrough, including why login coverage is deliberately checked twice, by both the verified database key and a pattern-matching egress rule, is on the account-takeovers incident analysis.

FAQ

Page-type database schema

Is page_types one field or 28 separate columns?

In the delivered schema it is one structured field, a map of up to 28 type=URL pairs. The free sample CSV represents it as a single page_types column; JSON and JSON Lines deliveries represent it as a nested object with one key per verified page type.

What happens when a domain has no verified URL for a given page type?

The absence is recorded explicitly rather than the key being silently omitted, so a consumer can distinguish "checked and not found" from "not yet checked."

Why does the schema carry both IAB v2 and IAB v3?

Different buyers and downstream systems standardize on different IAB taxonomy versions. Carrying both avoids forcing a migration on either group and lets a single database serve both integration paths.

Are Personas a fixed enumerated list or free text?

Personas are drawn from a defined set of more than 2,000 audience personas, assigned per domain based on who the site's content and structure indicate it serves.

Does num_distinct_page_types include absent types in its count?

No. It counts only the page types with an actual verified URL for that domain, from 0 to 28, so it functions as a classification-depth signal.

Where do I see this schema applied to real rows?

The free 100-domain sample CSV at /sample_databases/agent_allowlist_sample.csv uses this exact column order and field set on real, well-known domains.

See the schema on real rows.

Download the free 100-domain sample or review database license pricing.

Download sample CSV See pricing