AI Agent Allowlist
Free Sample
Home Page-Types Database Agent Guardrails 2026 Incidents API Docs Pricing
Resources
Use Cases Industries & Buyers Learn: Core Concepts Implementation Guides Comparisons Schema & Data Reference FAQ Glossary
Why It Matters
2026 Agent Incidents Category Targeting Database Refreshes Contact Customer Login
Download Free Sample
the canonical faq · 28 questions

Everything About AI Agent Allowlist, Answered in One Place

Product basics, data and schema, policy and enforcement, and pricing and licensing — the questions we hear most, organized into four groups so you can jump straight to what you need. Every answer links to the deeper page it summarizes.

28Questions answered
4Groups: basics, data, policy, pricing
40M+Domains the data behind these answers covers
$99Entry-tier API plan, 90,000 lookups/mo
Product basics 7 Data & schema 7 Policy & enforcement 7 Pricing & licensing 7

This page exists because the same handful of questions come up whether someone lands here from a search for pricing-page URL detection, a comparison of allowlists and blocklists, or a procurement checklist for an agent gateway. Rather than scatter partial answers across dozens of pages, this is the single place every question gets a complete answer, with a link to whichever deeper page covers the full argument. If you are new to the product, read Product basics first; if you already understand the pitch and are evaluating the data itself, start with Data & schema; if you are past evaluation and into rollout, Policy & enforcement and Pricing & licensing are where the practical decisions live.

Showing all 28 questions

01

Product basics

What this is, who it's for, and how it differs from adjacent tools like robots.txt and a WAF
What is AI Agent Allowlist?
A database of verified page-type URLs for 28 page types per domain across 40 million+ domains, built by analyzing 10 billion+ links. It is the policy data layer agent platforms, AI gateways, enterprise browsers, and forward proxies use to decide, before an autonomous agent's request leaves, whether a given URL may be visited or must be denied.
What problem does this actually solve?
Autonomous agents browse, click, and submit forms on their own. Without page-level policy, an agent hunting for a pricing page can just as easily wander onto a login form, a checkout flow, or an upload endpoint. This dataset lets a policy engine tell the difference between those pages on any of 40 million domains, so allow and deny decisions can be made per page type rather than per domain — the same agent can be allowed onto a vendor's documentation and pricing pages while every credential and transaction surface on that same domain stays denied. See why domain-level blocking fails agents for the fuller argument.
What are the four enforcement layers?
(1) The High-Value Host List, roughly 60 curated dangerous hosts such as cloud metadata endpoints. (2) The 40M-domain page-type database. (3) The Egress Rules Library, roughly 40 method-aware URL-pattern rules that catch risky shapes on any domain, including wiki edits, WebDAV writes, and plugin installs. (4) Default-deny for anything left unclassified. Evaluation order matters: a hard deny on the host list wins first, then a page-type match, then the pattern rules, and only requests that clear all three without a positive allow fall to the default-deny layer. A policy engine checks all four before a request leaves — the full model is on the agent guardrails page.
What is the difference between the API and the full database license?
The API is a per-lookup subscription starting at $99/month, ideal for prototypes and workloads under roughly 10M lookups a month, with data always current. A database license is a one-time on-prem purchase from $14,999, loaded into your own infrastructure with no per-lookup network calls and no data leaving your systems; add optional monthly refreshes to keep it current. Full request/response shape on the API overview.
Who actually uses this data?
Agent platform vendors, AI gateway and enterprise browser builders, forward-proxy and SASE/SSE vendors, and internal security teams running their own agent fleets — anyone who needs to decide, per URL, whether an autonomous agent may proceed. The buyer can be the team building the enforcement point itself, or a security team procuring data to plug into infrastructure they already run. See the industries and use-cases sections for specific deployment patterns.
What is a page type?
A classification of what a specific URL on a domain is for — login, pricing, checkout, documentation, and so on — rather than what the domain as a whole is about. A single domain can carry up to 28 distinct page types, each with its own verified URL, so the same domain can be simultaneously allowed on one page and denied on another depending entirely on which specific URL an agent is about to open — the same distinction that makes it possible to allow a vendor's pricing page while denying its login page without writing a single domain-specific rule. See the full taxonomy on page types, explained or the glossary entry.
How is this different from robots.txt or a web application firewall?
robots.txt is a site's own statement about how it wants to be crawled, and a WAF protects a site from inbound traffic. This is the mirror image: a policy an agent enforces on itself about which outbound destinations it may reach, independent of what any individual site's own rules say. Full comparison on robots.txt vs. agent policy.
02

Data & schema

Coverage, fields, verification method, and the free sample
How many domains does the database cover?
40 million+ domains, representing 99.99% of active internet usage, built by analyzing 10 billion+ links. Database license tiers run from a 10M-domain slice up to the full 40M+ repository.
What are the 28 page types?
Identity: login, signup, password_reset. Commerce: cart, checkout, subscribe, pricing, product. Content-write: post_create, comment, upload, community. Research/read: documentation, help_center, integrations, status, security, contact, about, leadership, careers, blog, press, events, partners, case_studies, legal, sitemap. Full field reference on the page-type schema page.
What are the 8 action types?
signup, password_reset, cart, checkout, upload, post_create, comment, and subscribe — pages where an agent does something (creates an account, submits a form, spends money, publishes content) rather than reads. They ship with the current reclassification cycle, so not every domain has them verified yet. See action type in the glossary.
What other metadata comes with each domain?
700+ IAB content categories (v2 and v3, Tier 1–4), a 59-category web-filtering taxonomy, language, country, global and country popularity rank groups, Open PageRank, and audience personas — all included in the same lookup as the page-type map, at no extra cost. This lets an agent policy combine a content-vertical question ("is this a financial services site") with a page-location question ("is this its login page") in a single record instead of two separate services.
How are URLs verified rather than guessed?
By traversing each domain's actual link structure and confirming a candidate page's real function, rather than assuming a path pattern like /login or /pricing exists. A URL only appears in the database if it was confirmed live on that domain; path-guessing heuristics fail across the platform and locale diversity of the long tail of the web, where a login page might live at /signin, an /account/login subpath, or an entirely separate identity-provider subdomain. Deep dive on verified URLs vs. guessed paths.
What does found: false mean in an API response?
The domain is not among the 40M+ classified domains. For a guardrail, that absence is itself a signal: an unvetted domain is a default-deny candidate rather than a silent allow, exactly the posture that keeps a policy safe against the long tail of the web it has not seen before. Full response schema on the API docs.
What's in the free sample CSV?
100 real, well-known domains with the full record: domain, page_types, language, IAB v2 and v3 Tier 1–4, Web Filtering Category, Personas, OpenPageRank, Country, Global and Country-Level Popularity Rank Group, and num_distinct_page_types. Download it here.
03

Policy & enforcement

Default-deny, the companion rule files, redirects, and the 2026 incidents
What is default-deny?
A policy posture where any request the check cannot positively resolve to an explicitly allowed page type is refused rather than permitted — because the domain is unclassified, the page type is not on the allow list, or the lookup itself failed. The alternative, default-allow, requires an ever-growing denylist that is always behind whatever new site an agent encounters next. See default-deny policy design.
What is the Egress Rules Library?
Roughly 40 method-aware URL-pattern rules that classify a URL on any domain before the request, covering shapes like wiki edits, WebDAV, plugin installs, and signup patterns. It ships with every database license and is evaluated server-side by the API. Full list on the Egress Rules Library page.
What is the High-Value Host List?
A curated list of roughly 60 dangerous or high-value hosts — cloud metadata endpoints, cloud consoles, package registries, tunnel sinks — checked ahead of the page-type database and rules layer, since a hard deny on this list wins regardless of what else matches. Unlike the page-type database, coverage here is intentionally small and hand-curated rather than exhaustive, because these are hosts worth denying on sight rather than evaluating case by case. It ships with every database license and API plan at no extra charge. Details on the High-Value Host List page.
How should redirects be handled?
Re-check the destination URL after every redirect before letting an agent's HTTP client follow it further. A URL that was checked and allowed does not stay allowed through a subsequent hop to a different page or domain — this is the single most common gap in naive implementations, and it is exactly the boundary a merchant's checkout hop to a shared payment gateway domain crosses, as covered on checkout page detection and the implementation guide.
Can an agent ever be allowed onto a checkout page?
Only under an explicit, narrow exception with a human-approval gate, never under a general research or browsing policy. checkout, cart, and subscribe are action types denied by default in nearly every sample policy on this site, and even a purchase-authorized agent should be scoped to a specific, pre-approved list of domains rather than any checkout page it happens to reach. See blocking agents from checkout and the one exception in agentic commerce safety.
How does this differ from prompt-based guardrails?
A prompt-based guardrail is an instruction the model may or may not follow under pressure, injected content, or a long context; a page-type check is a deterministic, pre-request control enforced outside the model entirely, so it holds even when the model itself has been talked into ignoring its own instructions. This distinction is also why prompt injection matters here: a hostile page's content might successfully convince an agent's reasoning to attempt a dangerous navigation, but the enforcement point outside the model still evaluates the actual URL against the same four layers regardless of what the model was persuaded to do. The two are complementary layers, not substitutes for each other. See deterministic vs. prompt guardrails and prompt injection URL defense.
What happened in the 2026 AI agent incidents, and would this have stopped them?
Several 2026 incidents involved agents reaching write or credential surfaces they should never have touched, including a Hugging Face dataset-upload breach, a DseWiki wiki-edit hijack, a JFrog Artifactory covert channel, and account takeovers across several services. Our analysis of the public disclosures shows the database and egress rules would have denied nearly every entry point, pre-request.
  Would your agents have been stopped? Check the incident-by-incident prevention analysis →
04

Pricing & licensing

API plans, database tiers, refreshes, OEM, and payment
How much does API access cost?
Pro is $99/month for 90,000 lookups, Pro Plus $249/month for 225,000, Advanced $499/month for 450,000, Advanced Plus $999/month for 900,000, and Business $1,997/month for 2,000,000 lookups at $1.00 per 1,000. Custom volumes are available beyond 10 million lookups a month. Full table on the pricing page.
How much do database licenses cost?
10M domains for $14,999, 15M for $24,999, and 30M for $49,999, one-time and perpetual. Optional monthly refreshes cost 30% of the license price per year. 40M+ and OEM cuts are available on request. See coverage tiers for which size fits which workload.
Should I choose the API or a database license?
Start on the API if you want to be live in minutes and your volume is under roughly 10M lookups a month; move to a database license if you need offline operation, sub-millisecond local lookups, or data residency guarantees that rule out an external call per request. Both return the same schema, so switching later is a data-source change, not a policy rewrite. Worked comparison on the API overview and build vs. buy.
Do refreshes cost extra?
Yes, and they are optional. A one-time database purchase is a snapshot frozen at delivery-date accuracy; the refresh add-on re-delivers your tier every month with re-verified classifications and newly screened domains for 30% of the license price per year. API plans are always current with no separate refresh charge, since each lookup hits the live database rather than a static snapshot. Whichever you choose, canceling a refresh subscription later still leaves you with your most recent delivered copy, permanently.
What is OEM licensing?
Custom licensing for products that redistribute this data inside their own shipping product — a secure web gateway, enterprise browser, or agent framework — covering redistribution rights, custom delivery formats (CSV, JSON, SQL, Parquet), and a refresh cadence matched to the licensee's own release cycle. Standard licenses cover internal use only; OEM specifically covers shipping the data to your own customers. See agent platform vendors for a worked OEM scenario.
Is there a free trial?
The free 100-domain sample CSV lets you inspect the real record structure and verification quality before any purchase. There is no time-limited trial of the paid API or database tiers; the sample is the recommended way to validate before subscribing, and it uses the exact same field names and structure the paid API and database deliveries use, so anything you build against it carries over unchanged.
What payment methods are accepted?
API subscriptions bill through PayPal, which also accepts credit and debit cards as a guest with no PayPal account required, monthly, cancel anytime. Database licenses can additionally be paid by bank transfer against an invoice, with annual invoicing available for Enterprise and OEM agreements on the enterprise page, including purchase-order and net-terms handling for procurement teams that require it.
Still exploring

Go deeper by topic

This FAQ answers the most common questions across every section of the site, but it deliberately stays at summary depth. Each topic below has a dedicated page that goes further: worked examples, code sketches, comparison tables, and the specific numbers behind a claim this FAQ only summarizes in a sentence or two. Use this page to get oriented fast, then follow a link when you need the full argument.

Concepts & definitions

Full glossary (40+ terms) What is an AI agent allowlist Allowlist vs. blocklist Agentic browsing risks Zero trust for AI agents Credential surface taxonomy

Use cases by role

Procurement research agents Price-monitoring agents Vendor due diligence Recruiting & sourcing agents Agentic commerce safety

Implementation guides

Reference implementation LangChain URL policy OpenAI Agents SDK guardrails MCP server access control Blocking agents from login

Comparisons

robots.txt vs. agent policy Manual curation vs. database Categories vs. page types Prompt guardrails vs. URL policy

Reference & product

The page-type database Agent guardrails, the four layers Full API documentation Checkout page detection Pricing page URL dataset Pricing & licensing AI Tools Blocklist (sibling product) Web Filtering Database (sibling product)

Didn't find your question?

Download the free sample to see the data firsthand, or contact us directly.

Download the Sample