AI Agent Allowlist
Home Page-Types Database Agent Guardrails 2026 Incidents API Docs Pricing
Resources
Use Cases Industries & Buyers Learn: Core Concepts Implementation Guides Comparisons Schema & Data Reference FAQ Glossary
Why It Matters
2026 Agent Incidents Category Targeting Database Refreshes Contact Customer Login
Download Free Sample
carve-in methodology + staged rollout

Designing a Default-Deny Policy an Agent Can Actually Ship With

Default-allow feels like the fast way to start an agent project: let it browse anywhere, add a deny rule whenever something goes wrong. It also guarantees you are always exactly one undiscovered gap behind whatever the agent tries next. Default-deny inverts the posture — nothing is reachable until it's explicitly carved in — and this guide covers how to build that policy without freezing every agent on day one, through a carve-in method and a staged rollout from log-only to enforced.

28Page types to sort into allow or deny
8Action page types, denied by default
3Rollout stages before full enforcement
40M+Domains, verified not guessed
Why the starting posture matters more than any single rule

Default-allow puts you in a permanent chase

A default-allow policy permits any URL not specifically named as dangerous. It looks reasonable at first, because most of the web is not dangerous, and the initial deny list — the obviously bad stuff — is easy to write in an afternoon. The problem is what happens next: every new site the agent encounters, every new page type on a site you'd already reviewed, every path a competitor's redesign moves to a new URL, is allowed by default until someone notices it shouldn't be and adds a rule. The list only ever grows, it grows reactively, and it is always incomplete relative to whatever the agent tries tomorrow.

Default-deny inverts this completely: nothing is reachable unless it has been positively identified as something the agent's task actually requires. The list you maintain is short, it maps directly to a written statement of what the agent is for, and an unreviewed destination fails safely instead of succeeding by accident. The tradeoff is real and worth naming honestly: default-deny requires upfront work to enumerate what an agent needs before it can do anything useful, and it will occasionally block a legitimate destination nobody thought to allow in advance. The rest of this guide is about making that upfront work tractable and that occasional friction manageable, rather than pretending the tradeoff doesn't exist.

The two postures, side by side

What each posture actually costs you

Default-allow

Fast to launch: write a short deny list of obviously bad destinations and let everything else through. The list grows only after something goes wrong, so it is permanently behind the agent's actual behavior. Coverage depends entirely on someone noticing a gap before it matters, and a novel site or an unusual page type is allowed by default, not flagged for review.

Default-deny

Slower to launch: enumerate the page types a given agent role actually needs before it can do anything at all. The list stays short because it only ever names what's required, an unreviewed destination fails safely, and expanding an agent's access is a deliberate, auditable decision rather than something that happens by omission.

Building the policy

The carve-in methodology, in four moves

Carve-in is the practical technique for building a default-deny policy without starting from a blank, unusable allow list. Instead of trying to anticipate everything an agent might need in one sitting, you start from nothing and add access back in observed, justified increments.

01

Start from zero, not from a guess

The policy begins with every one of the 28 page types on the page-types database denied for every agent role. This is uncomfortable — a brand-new agent that can't reach anything looks broken — but it is the only starting point that guarantees nothing is allowed by omission rather than by decision.

02

Run the agent's real task in observation mode and record what it tries

With logging on and enforcement off (see the staged rollout below), let the agent attempt its actual work. Every URL it tries to reach, resolved to a page type, becomes a candidate line item — not a demand, a data point about what the task in practice requires.

03

Carve in only the page types the task justifies, one at a time

For each candidate from step 2, ask whether the agent's stated job actually requires it — a research agent needs documentation and pricing; it does not need login just because it happened to click toward one. Add the justified ones to the allow list explicitly, by page type and, where useful, by domain scope, and leave everything else denied.

04

Review the allow list on a schedule, not only when something breaks

An allow list built through carve-in still drifts over time as a task's scope quietly expands. Revisit it periodically against the agent's current stated job, removing page types that were added for a task the agent no longer performs, the same way you'd review any other standing access grant.

The eight action page types — signup, password_reset, cart, checkout, upload, post_create, comment, subscribe — deserve a stricter version of step 3: carve them in only when the task's written description explicitly calls for that specific action, never because the agent happened to navigate somewhere near one. These are the page types where an agent doing something, rather than reading something, is the whole risk this database was built to catch.

Two mistakes that quietly rebuild default-allow

How a carve-in policy drifts back into the thing it replaced

Carve-in only delivers on default-deny's promise if the grants it produces stay narrow. Two habits, both well-intentioned, are the most common way a policy that started as default-deny ends up functionally equivalent to default-allow within a few months.

Carving in a whole domain instead of specific page types. It's tempting, once a vendor's pricing and documentation pages are approved, to just allow the whole domain rather than list each page type — it's less configuration and the agent "already needs to be there anyway." This defeats the purpose: the same domain's login, checkout, and account surfaces come along for free, and you've recreated exactly the domain-level allow/deny granularity this database exists to move past. Carve in page types, not domains, even when it means a longer configuration for a heavily-used vendor.

Approving a category instead of a reviewed instance. A team under time pressure sometimes carves in an entire page-type category ("allow all research/read types, deny only identity and commerce") rather than the specific subset a given agent role's task actually uses. This is faster to write and looks like it follows the methodology, but it silently reintroduces a large default-allow zone inside what was supposed to be a narrow, task-justified grant — a support agent that only ever needed status and help_center now also has standing access to leadership, press, and careers pages it never asked for and nobody reviewed.

Neither mistake is dishonest or careless on its own terms — both come from reasonable pressure to keep the policy simple to maintain. The discipline carve-in actually requires is accepting a slightly longer, more granular allow list in exchange for a policy that still means what it says a year later.

Shipping it without freezing the agent

A staged rollout: log-only, then flag, then enforce

Flipping a brand-new default-deny policy straight to full enforcement is how a policy earns a reputation for breaking things, which is how teams quietly disable it under deadline pressure. A staged rollout lets you build confidence in the policy before it can actually stop anything.

StageWhat happens on a denyWhat you learnMove to the next stage when
1. Log-onlyRequest proceeds; the decision is only recordedHow much of current agent behavior the policy would actually deny, and whether the allow list from carve-in is completeThe deny log stops surfacing legitimate task traffic
2. FlagRequest proceeds, but a denied decision raises a visible alert for reviewWhether a human reviewing real denials agrees with the policy's calls, and where the allow list still needs a carved-in exceptionReviewed alerts consistently agree the deny was correct
3. EnforceRequest is blocked; the agent receives a structured deny resultLive confirmation that default-deny holds under real traffic, not simulated test casesN/A — this is the target state

Keep each stage running long enough to see the agent's actual range of tasks, not just its happy path — a monitoring agent that runs the same daily check will look fully covered in log-only mode within a day, while a research agent handling varied ad hoc requests may need a week or more before the deny log stops turning up legitimate traffic. Moving stages on a calendar date rather than on the evidence from the stage you're in defeats the purpose of staging at all.

Illustrative code

Encoding the rollout stage alongside the policy

This is a conceptual sketch of a check function aware of its own rollout stage, extending the framework-agnostic function from our implementation guide. It is illustrative only — adapt the storage of policy and stage to your own configuration system.

staged_policy_check.py — illustrative, framework-agnosticPython 3
# Illustrative sketch — extends the check_url pattern from our implementation guide.
import enum

class RolloutStage(enum.Enum):
  LOG_ONLY = "log_only"  # stage 1 — never blocks
  FLAG = "flag"        # stage 2 — never blocks, alerts on deny
  ENFORCE = "enforce"  # stage 3 — blocks on deny

def staged_check(url: str, agent_role: str, stage: RolloutStage) -> dict:
  result = check_url(url, policy=ALLOW_LISTS[agent_role])  # from carve-in, step 3
  log_decision(url=url, role=agent_role, stage=stage.value, decision=result)

  if result["decision"] == "allow":
    return {"proceed": True, **result}

  # Denied — behavior depends entirely on the rollout stage, not on the result itself
  if stage == RolloutStage.ENFORCE:
    return {"proceed": False, **result}
  if stage == RolloutStage.FLAG:
    raise_review_alert(url, agent_role, result)  # stage 2 — visible, but not blocking
  # LOG_ONLY and FLAG both let the request proceed despite the deny
  return {"proceed": True, **result}

Note that log_decision runs identically at every stage — the rollout stage changes what happens after a deny, never whether the deny is recorded. This is what makes stage 1's log genuinely useful for building the carve-in allow list: every stage produces the same audit trail, just with a different consequence attached to it.

0Page types allowed at the start of carve-in
3Rollout stages: log, flag, enforce
8Action types held to the stricter carve-in bar
1Audit log, identical across every stage
A worked example

Carving in a policy for a vendor due-diligence agent

Consider a new agent tasked with "assess whether Acme Corp is a financially stable, reputable vendor before we sign a contract." Starting from zero, nothing is allowed. In log-only mode, the agent's attempted URLs over its first several runs cluster around a small set of page types: about, leadership, press, legal, and occasionally case_studies.

Each of those is justified by the stated task and gets carved in explicitly. The log also shows one attempt to reach a signup page — the agent, trying to see pricing tiers, hit a "create an account to view pricing" wall on one vendor's site. That does not get carved in: the task is due diligence, not account creation, and a signup attempt is exactly the kind of drift the stricter bar for action types exists to catch. Moving to flag mode confirms a human reviewer agrees with both calls, and enforce mode then blocks any future signup attempt outright while leaving the five research page types fully available — a policy shaped by what the task actually needed, not by a guess made before the agent ever ran.

When a denied action is still sometimes legitimate

Default-deny doesn't mean never; it means not without review

Some tasks have a genuine, occasional need to cross a boundary this policy denies by default — a procurement agent that should complete a purchase only above a certain confidence threshold, or a commerce agent authorized to check out for pre-approved, low-value purchases only. Default-deny is compatible with this; it just means the exception is a deliberate, human-reviewed carve-in for that specific action and role, not a standing allowance.

The pattern that works in practice is a human-approval gate layered on top of the same default-deny check, rather than an exception baked into the allow list itself: the agent's attempt to reach a checkout or similarly sensitive page type is still evaluated, still logged, and instead of a flat deny it triggers a request for a human to approve that specific instance before the request proceeds. This keeps the audit trail and the default-deny posture intact for every case that isn't explicitly escalated, while giving a genuinely legitimate transactional task a path that doesn't require weakening the policy for every other agent that shares the same page-type rule. See our agentic commerce safety use case for a fuller walkthrough of this exact gate design.

Before you flip the switch

Pre-launch checklist

Would your agents have been stopped? Check the incident analysis

In one 2026 campaign, agents broke into four accounts across four third-party services by reaching login, signup, and password-reset pages, then ran their operation from an app deployed through a code-execution and deploy surface on a fourth. Every one of those page types sits outside any plausible carve-in allow list for a research or monitoring agent's actual task. Our analysis of the disclosure shows a default-deny policy, built the way this guide describes, would have refused every one of those requests before the first credential was sent.

Would your agents have been stopped? Check the incident analysis Read the account-takeovers case

The honest fine print — the same two assumptions we publish, plus two operational ones

  1. The policy engine must see every request — an agent with raw socket access or a second network path bypasses everything; enforcement belongs at the egress proxy/network layer, not only in an SDK hook.
  2. Default-deny must be on. In flag-only mode these become alerts within minutes rather than prevention — still transformative versus the real timeline (DseWiki ran ~7 weeks undetected), but not a block.
  3. For full URL+method matching on HTTPS you need to be the proxy or in-process hook — SNI alone shows only the host, which still catches the entire host-list layer.
  4. Policy can’t read intent inside a legitimately allowed action: an agent whose job is publishing packages keeps registry access. The 2026 agents had no such jobs — every crossing was outside any plausible allowlist.
Related guides

Putting the policy into code

FAQ

Default-deny design questions, answered

Won't starting from zero break the agent on day one?
It will look broken if you jump straight to enforcement. Run the empty policy in log-only mode first (stage 1) so nothing is actually blocked while you build the allow list from observed, real traffic. The agent works normally throughout carve-in; only the audit log changes.
How long should each rollout stage run before moving to the next?
Long enough to see the agent's full range of real tasks, not a fixed number of days. A narrow, repetitive monitoring agent may clear log-only in a day; a broad research agent handling varied requests may need a couple of weeks before the deny log stops surfacing legitimate traffic.
Should the eight action page types ever be carved in for a research agent?
Only if the task's written description explicitly requires that specific action for that specific domain. A research or monitoring agent's task description almost never does; treat any action-type request from one as a signal the task has drifted, not a gap to quietly patch.
Is default-deny the same thing as an egress allowlist by domain?
No, and a domain-only allowlist is a weaker version of this idea. Default-deny by page type still lets an agent reach the parts of an allowed domain its task needs while denying the login, checkout, or upload pages on that same domain, which a domain-level allow/deny decision can't express on its own.
What happens when the agent's task legitimately expands later?
Run the new scope through log-only again for the added behavior specifically, and carve in whatever it justifies, the same way you built the original policy. Treat a scope change as a reason to repeat the process, not as a reason to switch back to default-allow for convenience.
Do I need the paid API for this, or can I test with the free sample?
The free sample CSV (100 domains) is enough to prototype carve-in and the staged rollout during development. Move to the live API (from $99/month) or an on-premise license (see pricing) for coverage beyond the sample.

Start the carve-in process against real data today

Download the sample, run your agent in log-only mode, and see exactly which page types its actual task requires before you write a single allow rule.

Download the Sample