Every access-control system makes one foundational choice: what happens by default. An allowlist denies everything except what is explicitly permitted; a blocklist permits everything except what is explicitly denied. For AI agents deciding, in real time, which of 40 million-plus domains to visit next, that single default matters more than almost any other design decision in the system — and the two approaches are not equally safe for this specific problem, even though both have legitimate uses.
Every destination is refused unless it matches an explicit rule that permits it. An unrecognized domain, an unclassified page, a URL nobody has evaluated before — all denied, by default, with no exception.
Every destination is permitted unless it matches an explicit rule that denies it. A new domain nobody has evaluated is allowed through by default, on the assumption that most of the internet is benign.
Notice the asymmetry in how each model fails. An allowlist that is missing an entry fails loudly — a legitimate agent gets denied somewhere it should have been allowed, and someone notices because the agent's task stalls. A blocklist that is missing an entry fails silently — an agent reaches a page nobody flagged as risky, and nothing visibly breaks until the consequence of that visit shows up later, if it ever does. That asymmetry, more than any specific feature comparison, is why default-deny is the standard starting posture recommended for autonomous agent browsing.
The security argument for default-deny is not specific to AI agents — it is the same reasoning behind firewall design, least-privilege access, and zero trust generally. Three points carry particular weight for agentic browsing specifically.
The space of possible destinations is unbounded and growing. An agent can be asked to research any vendor, check any competitor, or verify any integration, which means the set of domains it might reasonably need to visit is not a fixed list anyone can enumerate in advance. A blocklist has to anticipate bad destinations in a space that large; an allowlist only has to certify good ones, which is a smaller and more tractable problem when backed by comprehensive verified data.
The cost of a false allow is asymmetric with the cost of a false deny. An agent wrongly denied a legitimate research page produces an annoyance — a stalled task, a retry, a support ticket. An agent wrongly allowed onto a credential or transaction page can produce a real security or financial event. When the downside of one failure mode is categorically worse than the other, the default should tilt toward the cheaper failure, which is what default-deny does.
Attackers actively route around known-bad lists. A blocklist built from yesterday's known-bad domains does nothing against a freshly registered attacker-controlled domain, a compromised legitimate site, or an injected redirect to a page nobody has classified as dangerous yet (see our prompt injection URL defense guide for the mechanics). A default-deny posture does not need to have seen the specific bad destination before to refuse it — refusing the unrecognized is the whole point.
| Dimension | Allowlist (default-deny) | Blocklist (default-allow) |
|---|---|---|
| Handling of a brand-new, never-seen domain | Denied until verified | Allowed until proven bad |
| Maintenance burden | Requires a broad, current database (or a service that maintains one) | Requires only tracking known-bad entries — smaller list, but never complete |
| Failure visibility | Fails loud (false denial stalls a task) | Fails silent (false allow is invisible until exploited) |
| Best suited to | Open-ended browsing across an unbounded domain space | A narrow, well-understood, mostly-static set of destinations |
| Behavior against a novel attack (new domain, injected redirect) | Denied by default, no update needed | Allowed unless someone has already flagged that exact destination |
| Typical role in this product's four-layer model | The backstop — default-deny for anything unclassified | The host list and egress rules — curated deny-by-shape on top of the backstop |
Default-deny is the right general-purpose posture for open-ended agent browsing, but it is not universally correct. A few concrete situations favor a blocklist-style approach, and it is worth naming them honestly.
An agent that only ever needs to reach a handful of internal or pre-approved partner systems, known in full at build time, gets little benefit from a 40-million-domain database and may reasonably run a small, explicit allow-and-deny list instead — which is itself a form of allowlist at a smaller scale, not really a blocklist.
An agent restricted to a company's own internal wiki or documentation platform, with no ability to leave that platform at the network layer, has a much smaller and better-understood space to defend — the risk calculus that favors default-deny for open internet browsing applies less forcefully here.
During early development, before an organization has decided what its actual agent policy should be, a temporary blocklist of obviously dangerous destinations can unblock experimentation faster than standing up full default-deny enforcement — with the explicit understanding that it is a stopgap, not the production posture.
Each of these cases shares a common thread: the space of destinations the agent can reach is small, known, or otherwise constrained by something other than the URL policy itself. As soon as that constraint is lifted — the agent gains broader internet access, the fixed list grows, the prototype goes to production — the case for switching to default-deny gets substantially stronger.
Procurement, competitive intelligence, market research, and lead-generation agents are, by design, meant to reach domains nobody specified in advance. That is exactly the situation a blocklist cannot defend, because the domain in question may be completely new to everyone's knowledge.
The moment an agent's toolset includes submitting a form, creating an account, or completing a purchase, the cost of a single false allow rises sharply. Identity and transaction page types should be denied by default regardless of domain, which is a default-deny posture almost by definition.
Banking, insurance, government, and legal workflows typically need to demonstrate what an agent was prevented from doing, not just what it did. A default-deny posture with a full audit log (see our navigation audit log guide) produces that evidence naturally; a blocklist produces evidence only for the specific things someone thought to block.
It would be tidy to say "always use an allowlist, never a blocklist," but that overstates the case and is not, in fact, how our own four-layer model works. A precise description matters more than a clean slogan.
Layers 1 and 3 are, structurally, blocklists: specific, curated things to deny, evaluated against a request regardless of which domain it targets. What makes the overall system an allowlist in the meaningful sense is layer 4 — nothing gets through by default just because layers 1–3 did not happen to flag it. The blocklist-shaped layers exist because some risky patterns (a cloud metadata endpoint, a wiki-edit URL shape) are worth catching by pattern on any domain, even ones the page-type database has already classified as broadly safe. The allowlist backstop exists because the alternative — assuming anything not on a curated deny list is fine — is precisely the failure mode the theory section above argues against.
Abstract arguments about defaults are easier to evaluate against one concrete task, run twice under each model.
Suppose a market-research agent is given the task "find and summarize the pricing pages of the top five competitors in a given software category," with no pre-specified list of which five companies that will turn out to be. Under a blocklist policy, the agent is free to reach any domain it identifies as a plausible competitor, checked only against a curated list of known-bad domains and page shapes. In the common case, this works fine: the agent finds five real companies, reaches their pricing pages, and reports back. The blocklist adds essentially no friction, because none of the five happen to appear on the known-bad list.
Now suppose one of the five domains the agent identifies is a recently registered lookalike of a real competitor's brand, set up to harvest visitor information, or a legitimate competitor's domain that has a compromised subdomain hosting a credential-phishing page under an innocuous-looking path. Neither would appear on any blocklist assembled before the task ran, because neither existed, or neither was known to be malicious, at the time the list was last updated. Under the blocklist policy, the agent reaches the page and processes whatever it finds there, exactly as if it were legitimate. Under a default-deny allowlist policy, the same request is evaluated against the page-type database and egress rules rather than against a static bad-domain list: if the destination is unclassified, or if the specific path shape matches a rule for credential harvesting or an identity page type, the request is denied regardless of whether anyone had previously flagged that exact domain as dangerous.
The blocklist policy performed identically to the allowlist policy in the common case and diverged sharply in the uncommon one — which is precisely the case a security review cares about most, and precisely the case a list of known-bad domains, by construction, cannot anticipate. This is the theoretical asymmetry from earlier in this page showing up as a concrete outcome: the extra friction of default-deny on the 95% of requests that were always going to be fine buys protection on the 5% that were not, and that 5% is unevenly distributed toward exactly the destinations worth worrying about.
The most common objection to default-deny in practice is not theoretical — it is operational: "a blocklist is cheaper to maintain than a database broad enough to cover legitimate destinations." That objection has real substance and deserves a direct answer rather than a dismissal.
It is true that a hand-maintained list of a few hundred known-bad domains costs less to build than a 40 million-domain, 28-page-type database verified from live link structure. But that comparison leaves out where the maintenance cost actually goes under each model. A blocklist's maintenance cost is paid continuously and invisibly, in the form of incidents involving destinations nobody added to the list in time — a cost that does not show up on an engineering roadmap because it looks like a security incident, not a maintenance backlog. An allowlist's maintenance cost is paid up front and visibly, in the form of a database or subscription that needs to stay current as sites restructure, which is why refresh cadence is treated as a real product feature (see the freshness discussion on the page-types database page) rather than an afterthought. Comparing the sticker price of a static list against a live, maintained database is comparing a cost that has been deferred against one that has been made explicit — not comparing two costs of similar shape.
Several 2026 agent incidents involved destinations — a specific Hugging Face upload path, a specific wiki's edit endpoint, a Modal-hosted app — that a blocklist assembled in advance would have had no reason to include. Our analysis shows the default-deny backstop, not the curated deny lists alone, is what closes that gap; would your agents have been stopped by a list built only from yesterday's known threats?
Would your agents have been stopped? Check the incident analysis Read the JFrog Artifactory caseThe honest fine print — the same two assumptions we publish, plus two operational ones
The foundational definition this page builds on.
What an allow or deny decision is actually made against.
The product view of the four-layer hybrid described above.
The ~40 blocklist-shaped rules layered on top of default-deny.
A genuine blocklist product, for a genuinely different problem: humans reaching AI tools.
Download the sample CSV and see exactly which page types resolve to allow versus deny across 100 real domains.
