The technical half of the site. What a record contains, how the coverage tiers map to real agent traffic, how login and checkout URLs are actually found, and what the lookup API returns.
Each domain carries its verified page-type URLs plus the classification columns: IAB v2 and v3 down to tier 4, a 59-category filtering taxonomy, language, country, popularity rank groups, OpenPageRank and audience personas. The sample CSV is this exact schema.
# one row, abbreviated domain "stripe.com" page_types login, pricing, documentation, status, careers, ... num_distinct_page_types 17 IAB v2 Tier 1-4 Technology & Computing > ... IAB v3 Tier 1-4 Business and Finance > ... Web Filtering Category Finance Personas Developers, Finance Professionals OpenPageRank 7.8 Language / Country en / US Global / Country Rank Group Top 10K / Top 5K
Every column in the database documented one at a time: domain, the 28 page types, IAB v2 and v3 tiers, the filtering category, personas and popularity rank groups. Start here if you are mapping the data into your own model.
How each coverage tier maps to real agent navigation traffic, and how to size the right one against a workload's domain diversity rather than against a headline number.
How login URLs specifically are found and confirmed, plus the SSO, subdomain and locale edge cases that make guessing /login unreliable in practice.
How the database distinguishes the three, why they carry different risk levels, and why a redirect to a payment gateway domain needs a fresh policy check rather than inheriting the last verdict.
Pricing is the single most-requested lookup in the dataset. How country and currency locale variants are handled, and what that means for a monitoring agent hitting the same domain from different regions.
A plain-language overview of the endpoint, the response shape, authentication and rate limits, illustrated on the $99/month Pro plan. The full technical docs live separately.
The questions that come up in evaluation: what verified means, what a one-time license includes, refresh cadence, fair use on the API, and what happens to domains that are not in the database.
Every term used across the site defined in a few sentences and anchor-linked, from credential surface and default-deny through to egress rule and page type.
One-time, perpetual, on-premise. The database sits inside your perimeter with no calls back to us. Optional refresh runs at 30% of the license price per year.
Prefer live lookups to a local copy? The API starts at $99/month for 90,000 lookups. Full tiers on the pricing page.
The 2026 incidents are the clearest demonstration of why these columns exist. The Hugging Face breach, the DseWiki wiki hijack, the JFrog Artifactory covert channel and a run of third-party account takeovers all turned on an agent reaching a specific page type it had no business reaching. Our analysis shows the documented attack steps in almost all of them would have been denied at the request.
How the Hugging Face breach could have been stoppedTwo companion datasets ship with every database license and are documented separately: the Egress Rules Library, around 40 method-aware URL-pattern rules that classify a URL on any domain, and the High-Value Host List, roughly 60 curated dangerous hosts. For the broader filtering taxonomy at 100M+ domain scale, see the sibling product webfilteringdatabase.com.
Reference pages describe what the data is. These sections cover whether to license it, how to integrate it, and what to point it at.
Policy design, framework hooks, engine architecture, audit logging, data evaluation.
Compare — 8Build vs buy TCO, and honest comparisons against proxies, robots.txt and prompt guardrails.
Use cases — 15Fifteen agent jobs with the page types each one needs and the ones it must never reach.
