A bank deploying research or support agents cannot start from “allow everything, review later.” The regulatory posture is the reverse: every navigation an agent makes needs to be explainable after the fact, and every credential or transaction surface needs to have been refused before the fact. AI Agent Allowlist supplies the verified page-type data a default-deny policy needs to be precise instead of paralyzing — 28 page types across 40 million+ domains, so “deny” does not have to mean “deny the whole internet.”
This page is written for compliance, risk, and security engineering teams evaluating a data source for agent egress policy — not investment advice, and not a claim of regulatory certification. What follows is a description of the data and how institutions typically structure policy around it; your own compliance function determines what your regulatory obligations actually require.
Your customers were exposed to this. The 2026 agent incidents included agents breaking into four third-party accounts — the exact credential-surface failure a bank cannot tolerate at any scale. Our database and egress rules would have denied the login and account-creation requests before they reached those accounts.
A blocklist approach requires someone to have already identified a URL as dangerous before it is stopped — structurally reactive, and unacceptable when the downside is a compromised customer account or an unauthorized transaction. Default-deny inverts that: nothing is reachable until it is affirmatively classified as a safe page type on a domain within the bank’s risk appetite. This database exists to make that inversion practical rather than crippling — supplying the verified “this is safe” classifications a default-deny policy needs at the scale of 40 million+ domains, so research and support agents keep working while credential and transaction surfaces stay closed.
The distinction matters because a bank cannot simply refuse to deploy agents and call the risk closed — competitive pressure and internal efficiency demands are pushing agent adoption into research, support, and due-diligence workflows regardless. The realistic choice is not agents versus no agents; it is a default-deny policy built on precise, verified classification versus an ad hoc one built on hope and a system prompt.
A workable default-deny policy still needs a research agent to function. The table below is a representative starting policy for a financial-services deployment — institutions set their own final rules and risk tolerances.
The reason a page-type-driven policy differs from a naive “deny everything” posture is precision: default-deny only stays workable if the affirmative allow list is granular enough to cover the actual research surface an agent needs, and coarse enough that nobody has to hand-maintain a per-domain exception list. Page type is the unit of granularity that satisfies both constraints at once — a single rule (“allow documentation and pricing pages”) covers the research surface across all 40 million+ domains in the database, without requiring a compliance analyst to pre-approve each vendor domain individually before an agent can read its public-facing pages.
| Page type | Typical bank policy | Rationale |
|---|---|---|
| login, signup, password_reset | deny, always | Credential surfaces — no agent role justifies unattended access to any account, internal or third-party |
| checkout, cart | deny, always | Unauthorized transactions are a direct financial-control failure, not a browsing error |
| post_create, comment, upload | deny, always | Public-facing write actions carry reputational and disclosure risk with no compliance upside |
| subscribe | flag → approval | Low individual harm, but an unreviewed opt-in is still an unauthorized action taken in the institution’s name |
| pricing, documentation, integrations | allow, logged | Vendor research and due diligence, the core function most financial-services agents are deployed for |
| about, leadership, legal, security | allow, logged | Entity resolution and counterparty due diligence — read-only, no side effects |
| product, case_studies, press | allow, IAB-conditional | Allowed on approved verticals; deny or flag on high-risk IAB/filtering categories such as gambling or unregulated lending |
Banking compliance policy is rarely uniform across jurisdictions — what a research agent may read for a US retail-banking use case differs from what is appropriate for an EU-regulated entity or a sanctioned-jurisdiction screen. Two fields in every record exist for exactly this reason.
A multinational institution running the same agent framework across several regulated entities needs one dataset that supports different policies per jurisdiction, not several separate datasets. Country and language fields let a single licensed database back a US retail-banking policy, an EU-regulated-entity policy, and a sanctioned-jurisdiction screen simultaneously, each expressed as a different set of conditions over the same underlying records rather than as separate procurement efforts.
Every domain record carries a country association, letting policy differ by jurisdiction — for example, treating research on domains registered in sanctioned or high-risk jurisdictions as flag-or-deny regardless of page type, without a manual per-domain jurisdiction lookup.
Useful for routing research agents to the correct locale of a multinational counterparty’s site, and for compliance reviews that need to know what language a flagged navigation occurred in.
700+ IAB categories and 59 filtering categories let policy exclude entire verticals — unregulated lending, gambling, cryptocurrency exchanges outside an approved list — independent of page type.
Every record carries a last-processed timestamp, so a compliance review can establish exactly how current the classification was at the time of a specific logged decision, without having to reconstruct that from separate change logs.
Full database licenses run entirely inside your infrastructure with no per-lookup call to an external service — relevant for institutions whose data-residency policy restricts what may call out to a third party, or whose vendor-risk process requires minimizing external dependencies on the critical path of a control.
Because every navigation resolves to a page type, IAB category, and filtering category before a decision is made, the resulting log is structured evidence, not a narrative summary written after the fact, and it stays consistent across every agent role and every model version your institution deploys.
Consider a mid-size bank deploying an agent to support its third-party risk team: research prospective payment-processing vendors, pull background on leadership, and flag anything unusual before a human analyst opens a formal review. The agent needs broad reading access across the open web and zero ability to take action on the bank’s behalf.
On a candidate vendor’s domain, the agent reads about, leadership, and security pages freely — each allowed and logged, feeding directly into the analyst’s briefing. When a task description leads the agent toward what looks like a partner-portal link, the resolved page type comes back as login — denied immediately, logged with the reason, and the agent moves on to the next research task instead of attempting authentication it was never authorized to perform.
The vendor also happens to operate a customer community forum on the same domain. The agent’s task never asked it to participate, but if a poorly worded instruction ever pointed it at the forum’s posting form, the resolved page type post_create denies it by default — the same rule that would apply on any of the 40 million+ domains in the database, not a rule someone had to write specifically for this vendor.
Most financial institutions already run web-filtering for their human workforce — blocking gambling, adult content, and known-malicious domains. That stack is necessary and does not address agent-specific risk, for a simple structural reason: it was built to keep employees away from bad neighborhoods, not to govern what an automated system does inside neighborhoods everyone agrees are fine.
The counterparty domains a due-diligence agent researches are, almost by definition, legitimate businesses — exactly the sites a category filter is built to allow, not block.
An employee browsing a vendor’s site makes deliberate, accountable choices. An agent executing a task can be redirected by ambiguity or a hostile page toward an action no human ever approved.
An examiner asking about agentic-AI risk management wants to know what stops an unauthorized action, not which domains are on a blocklist — a page-type control answers the question actually being asked.
Separate research/support agents from any agent with a legitimate transactional mandate — they get different default policies from the start.
Log every would-be decision against real traffic before enforcing, to catch false positives against the specific vendors and counterparties your teams already research.
Turn on hard denial for identity, commerce, and content-write page types first — the categories with no ambiguity and the highest downside.
Subscribe, upload, and any jurisdiction-flagged navigation route to a human reviewer rather than an automatic allow or deny.
Institutions with strict data-residency and third-party-dependency policies typically license the full database for on-prem deployment rather than call an external API for every navigation, keeping every lookup inside the institution’s own network boundary.
| License | Coverage | One-time price | Refresh |
|---|---|---|---|
| Database — 10M domains | Top 10M by real-world usage | $14,999 | 30% of price / yr |
| Database — 15M domains | Extends into regional/niche properties | $24,999 | 30% of price / yr |
| Database — 30M domains | Near-complete active-web coverage | $49,999 | 30% of price / yr |
| 40M+ / custom slice | Full repository, or geography/vertical cut for regional coverage | Contact | Contact |
| OEM / MSSP redistribution | Embedded for a managed-service or multi-entity deployment | Custom | Custom |
Quarterly refresh is worth budgeting as a compliance line item, not a discretionary add-on: a stale classification is a control that silently degraded, and roughly 300,000 newly registered domains are screened per cycle — new counterparties and vendor sites your research agents will inevitably encounter. Full self-serve terms are on pricing; large-institution and OEM terms via contact.
For multi-entity groups — a holding company with several regulated subsidiaries, or a bank operating across multiple jurisdictions under separate charters — a single database license typically covers deployment across the group, with per-entity policy configuration layered on top of the same underlying classification data. This is worth raising directly during procurement, since it can materially change the economics compared to licensing separately per entity.
A regional bank's third-party risk team deploys an agent to pull public disclosures, leadership pages, and regulatory-filing links for new vendors entering the onboarding queue. The immediate objection from the second line of defense: what stops this agent from wandering onto a vendor's client-login portal, or a payment page that happens to sit on the same domain as the vendor's public disclosures?
The scoped answer is a policy that allows the read-only page types — about, leadership, press, legal, case studies — on domains named in the onboarding queue, denies every identity and commerce page type on every domain without exception, and logs every navigation with its page-type classification for the risk file. That log becomes the artifact the second line reviews before the vendor clears onboarding, in place of a written assurance that the agent "was told" to stay on public pages.
If your institution licenses this through a managed security provider rather than directly, the packaging discussion on AI gateway URL policy or an MSSP-delivered model may apply to your deployment. For the underlying four-layer mechanism — page types, egress rules, host list, default-deny — read agent guardrails and the full page-types database reference. Institutions also screening human employee access to unapproved AI tools may want AI Tools Blocklist as a companion control for that separate risk surface.
Verified page types, jurisdiction fields, and a logged decision on every navigation. Start with the sample, then talk deployment.
