Most teams start agent access control with a spreadsheet: fifty domains, a few dozen approved paths, updated when someone remembers. That approach has a decay rate. This page works through it against a continuously re-verified 40-million-domain page-type database.
A hand-list only covers what someone remembered to add. Several 2026 agent incidents, including the Hugging Face breach, the DseWiki hijack and the JFrog Artifactory covert channel, ran through domains no manually curated list would have flagged in time. See how the database and egress rules would have prevented almost all of them: the 2026 agent incidents, prevented.
Almost every team that deploys its first agent starts here, and there is nothing wrong with starting here. The failure mode shows up later, at scale, when nobody notices the list stopped being true.
The pattern is consistent across procurement-research agents, sales-intelligence agents, and internal automation tools alike: someone with security responsibility sits down, thinks through the agent's intended job, and writes down every domain they can think of that the agent legitimately needs. This is a reasonable first pass and it is usually correct on day one. The problem is not the initial list; it is the absence of any mechanism that keeps the list matching reality as the agent's actual behavior, and the internet underneath it, keep changing.
None of these require an attacker. They are what normal internet churn does to a static list, on a normal week, without anyone doing anything wrong.
It helps to be precise about what "stale" means here. A hand-list is not a snapshot that ages uniformly, the way a photograph fades. It decays unevenly: a handful of entries stay accurate indefinitely because the vendor's site structure never changes, while others go wrong within days of a redesign. The unevenness is what makes staleness hard to catch by spot-checking — the entries you happen to glance at are disproportionately likely to be the stable ones, because unstable entries are the ones tied to actively changing products.
An agent doing vendor research, price monitoring, or lead enrichment encounters domains the list's author never anticipated. Each one is either silently allowed by a permissive fallback, or the agent stalls waiting on a human to add it. Neither is a policy; both are gaps that widen every week the agent runs.
A vendor migrates their pricing page behind a new subdomain, adds a login gate to what was a public docs page, or merges their checkout flow into their account area. The hand-list still points at the old URL. Nobody re-verifies it, because re-verifying fifty domains by hand does not scale as a recurring task.
The list was built for one agent doing one job against a known set of partners. The next agent, doing a different job against a different long tail of domains, either inherits an irrelevant list or starts its own from scratch. Coverage never compounds; each new agent restarts the curation debt at zero.
A composite, realistic timeline built from the failure patterns above, not a specific customer account.
A procurement-research agent is scoped to 40 vendor domains, each with a manually noted pricing and docs URL. Checkout and login are hard-denied by domain, which works because the list is small enough to eyeball.
Real vendor research surfaces competitors and sub-vendors nobody pre-approved. The fallback policy is either "block everything not listed," which starves the agent of useful research, or "allow anything not explicitly denied," which reopens exactly the exposure the list was built to close.
One of the original 40 domains moves its pricing page from a marketing subdomain to its app subdomain, next to its billing and account pages. The hand-list still points at the old marketing URL. The new URL, unclassified, falls into whatever the fallback policy is.
An agent visiting a newly encountered partner domain reaches a page that looks like documentation but sits one click from an account-settings flow. Nothing in the hand-list said anything about this domain, because nothing in the hand-list could have anticipated it.
The team now faces the same curation problem at a larger scale, with less confidence in the existing 40 entries than they had in week 1. This is the point at which most teams evaluate a licensed database instead of a fourth rewrite of the spreadsheet.
The comparison practitioners actually need, not a marketing checklist.
| Dimension | Hand-maintained list | AI Agent Allowlist database |
|---|---|---|
| Initial coverage | 30-80 domains, whatever a person could enumerate | 40 million+ domains, 99.99% of active internet traffic |
| Page-type granularity | Usually just "allowed domain" or a hand-noted URL or two | 28 verified page types per domain, not guessed paths |
| New-domain coverage | Zero until someone files a ticket and adds it | Already covered if the domain is in the 40M repository |
| Stale-URL detection | None; entries rot silently | Refresh cycles re-verify against live link structure |
| Cross-domain risk patterns | Not covered; each domain is a one-off entry | ~40 egress rules catch risky URL shapes on any domain, listed and named |
| Ongoing labor | Recurring manual review, scales linearly with domain count | Optional monthly refresh at 30% of license price per year |
| Cost structure | Hidden in engineer time, rarely tracked | Fixed one-time license: 10M domains $14,999, 15M $24,999, 30M $49,999 |
| Fallback for unknowns | Ambiguous; usually defaults to allow by accident | Explicit default-deny for unclassified destinations |
A small, genuinely fixed set of partner domains does not need a 40-million-domain database behind it. The honest test is whether the domain set is closed and stable, or open and growing.
You do not need precise figures to see whether your deployment is on the wrong side of this trade. A simple model, using your own rough numbers, usually settles the question in a few minutes.
Start with three inputs you can estimate for any agent deployment: the number of distinct domains the agent visits in a typical week that were not on the original hand-list (call this the discovery rate), the number of entries on the current list that reference a URL last verified more than a quarter ago (the staleness count), and the number of person-hours it actually takes someone to research, verify and add one new domain entry correctly, including the follow-up when the first attempt gets a page type wrong (the unit cost). Multiply the discovery rate by the unit cost and you have a running weekly labor bill that never shows up on an invoice, because it is absorbed as "someone's Tuesday afternoon." Add the staleness count, because every stale entry is either silently wrong or requires the same unit cost to re-verify.
Illustrative example, using round assumptions rather than a claimed measurement: a team estimates 6 new out-of-scope domains a week, 15 minutes of verification-and-entry time per domain once someone actually gets to it, and roughly 10 stale entries accumulating per quarter that eventually need the same treatment. That is 90 minutes a week of pure list-maintenance labor before anyone accounts for the entries that get missed entirely because nobody had 90 minutes that week. Multiply by the number of agents in the fleet, each with its own list or its own carve-outs from a shared one, and the labor line item compounds faster than most teams expect when they first sketch the model.
Compare that running cost against a fixed one-time license. At the 10M-domain tier ($14,999 one-time), a team spending even a modest number of hours a month on manual list maintenance crosses the license price within a year or two purely in labor terms, before counting the incidents a coverage gap could cause. The database does not eliminate maintenance entirely — the optional refresh at 30% of license price per year exists precisely because domains change — but it converts an open-ended, linearly scaling labor cost into a fixed, budgeted one that does not grow every time an agent visits a domain nobody anticipated.
Replacing a hand-list with the database does not remove the need for the other three layers; it is what makes the other three effective at scale.
A curated list of roughly 60 high-value hosts — cloud consoles, registries, tunnel sinks — denied on any domain regardless of page-type classification. See the high-value host list.
The 40M-domain page-type database itself, replacing the hand-list with continuously verified coverage. See the full page-types database.
Roughly 40 egress rules matching risky URL patterns on any domain, including ones the database has not classified yet. See the egress rules library.
Default-deny for anything the first three layers did not resolve. No hand-list, however careful, offers a real fallback for the unknown.
A forward proxy sitting in front of your agent traffic can enforce these layers, but it needs page-type data to check against; see how the two pieces divide the work on proxy filtering vs. the agent allowlist. For a general framing of default-deny design, see agent guardrails. If the domains your agents need to reach are humans' AI-tool destinations rather than agent-navigation targets, the sibling product AI Tools Blocklist covers that separate use case with its own 20,000+ AI-tool domain dataset.
None of this is about competence. Teams that build careful hand-lists are usually the same teams that eventually replace them; the delay is structural, not a failure of judgment.
The first reason is that a hand-list's failures are invisible by default. A domain that should have been denied and was not does not announce itself; it just works, quietly, until an incident review or an audit asks why an agent was ever able to reach it. Compare that to a broken build or a failed deploy, which announces itself immediately and gets fixed the same day. Security debt of this shape survives precisely because nothing forces attention to it.
The second reason is sunk cost dressed up as familiarity. A team that has spent months tuning a 60-domain list knows its quirks, knows which entries are aliases for the same vendor, knows which ones were added defensively after a near-miss. Replacing it feels like discarding institutional knowledge, even when that knowledge is exactly the kind that does not scale past the person who holds it. The database does not discard that knowledge so much as make it unnecessary to hold in one person's head.
The third reason is that the switch is usually proposed at the wrong time: after an incident, when the team is in triage mode and adding a new dependency feels risky, rather than during a quiet period when evaluating a sample dataset and a pricing tier is a calm, low-stakes decision. Evaluating the free 100-domain sample CSV against your own agent's actual traffic log, before anything goes wrong, is the version of this decision that takes an afternoon instead of a postmortem.
Yes, and many teams do exactly this. The migration point to watch for is when the domain set your agents touch stops being predictable in advance. A free 100-domain sample CSV is available to test schema fit before licensing.
No. The database resolves page types on domains it has classified; the egress rules library catches risky URL shapes, like wiki-edit endpoints or deploy hooks, on domains the database has not seen yet. They are complementary layers, not substitutes.
There is no fixed answer, which is itself the problem: review cadence for a hand-list is usually reactive, triggered by an incident or a broken agent run, rather than scheduled. The database's refresh cycle is scheduled and does not depend on someone remembering.
Drift is the growing gap between what a list says about a domain and what is actually true on that domain today, caused by site redesigns, new subdomains, and domains entering scope that were never in the original list.
Licensing is one-time and tiered starting at 10M domains for $14,999, so a smaller deployment is not forced into the largest tier. See coverage tiers for how to size the tier to actual workload diversity, not domain count alone.
The full field-by-field reference, including page types, IAB categories, filtering category, and popularity ranks, is on the page-type schema reference.
Download the free 100-domain sample or review license pricing for the full 40M-domain database.
