Agencies piloting AI agents for research, records lookup, or constituent-facing automation face a procurement question a policy memo cannot answer: how does the agent know, at the exact moment of a request, which web page it may enter? AI Agent Allowlist answers it with a verified, purchasable database of page types and an egress rule set an evaluator can score against an RFP line item — delivered as a one-time, on-premise license with no calls back to us.
Most agency AI pilots start narrow: an internal research assistant that fetches vendor documentation, a records-lookup agent that reads public filings, a constituent-facing bot that checks a status page. The demo works because someone hand-picked the ten domains it touches. Procurement, general counsel, and the agency CISO are asking a harder question before anything reaches production: what happens on domain number eleven, the one nobody hand-picked, when the agent decides on its own that a login form or a payment page looks like the fastest way to finish its task?
That question does not have a good answer if the only control is a system prompt telling the agent what not to do. Instructions are negotiable inside the model, and a hostile or merely confusing page can push an agent past them. The fix that satisfies procurement, security, and legal at the same time is a control enforced outside the model — a lookup against verified data, evaluated before the request leaves the agency's own infrastructure.
A one-time license delivered as CSV, JSON, or an importable SQL dump. It sits inside your network boundary; nothing about a lookup leaves the agency, because nothing calls out to us.
Login, signup, password reset, checkout, and cart are page types in the schema — not guesses. An evaluator can point at a specific data field and ask "is this denied," and get a specific answer.
The egress rules and host list ship as reviewable files, not a black box. Legal and security staff can read every rule before signing off, and re-read it after every refresh.
We use the phrase deliberately narrowly. It does not claim a certification we do not hold — we are not asserting SOC 2, ISO, or FedRAMP status, and an evaluator should not take it to mean that. It describes three concrete properties of the delivery model that a procurement process can verify directly.
The on-premise license is a snapshot delivered as a file. It runs against your own lookup infrastructure with no outbound call to aiagentallowlist.com at query time. If our service is unreachable, your policy engine still evaluates every request — because it never asked us in the first place.
The page-type map, the roughly 40 egress rules, and the roughly 60 curated high-value hosts are delivered as files your security team can open, diff against the prior release, and sign off on before deployment — the same review discipline agencies already apply to firewall rule sets.
A one-time license stays exactly as delivered. If your program wants re-verified data on a cycle, the monthly refresh add-on is priced and optional per tier on the pricing page — it is never pushed to you without a separate line item.
These map directly to fields and files in the delivery, not marketing language. Score a vendor response against each one specifically.
Ask for the discovery method. Ours traverses each domain's own link structure; a page type is recorded as absent, never guessed, when a domain does not serve it.
Ask whether the vendor can deliver a file, not only an API. A metered API is a legitimate architecture for some programs, but a hard data-residency requirement needs a purchasable, hostable copy.
Ask for the exact page types in the schema and which ones the vendor recommends denying by default. Login, signup, password_reset, checkout, cart, and upload should be on that list.
40 million domains sounds arbitrary until it is tied to a coverage number: 99.99% of active internet usage. Ask any vendor quoting a domain count to also quote the usage percentage.
Stale credential-surface data is a governance gap in slow motion. Ask what changes between refresh cycles and how many newly registered domains get screened per cycle.
Not every workload needs on-premise delivery, and the API is a reasonable starting point for a pilot. But once an agency's data-residency policy or its network-boundary rules apply to a workload, a service that must be called for every navigation decision stops being a viable architecture — regardless of how good the answer is.
Coverage is measured in real-world traffic share, not raw domain count — the top tiers already capture almost all of it, and the difference between them is long-tail depth: regional sites, niche vendors, and newly registered domains an agency's agents might still encounter.
| Delivery | Coverage | Data leaves agency network? | Price |
|---|---|---|---|
| 10M-domain on-prem license | Majority of real-world traffic | Never | $14,999 one-time |
| 15M-domain on-prem license | 99.99% of active usage | Never | $24,999 one-time |
| 30M-domain on-prem license | 99.99%+ plus long-tail/regional | Never | $49,999 one-time |
| Monthly refresh add-on, any tier | Re-verified each cycle | Never (delivered as a file) | 30% of license price / year |
| Lookup API, self-serve | Same schema, hosted by us | Yes, per lookup | from $99/mo |
Every on-premise tier ships with the egress rules library and the high-value host list included, so credential, payment, and content-write surfaces are covered on domains outside the licensed page-type map too. 40M+ and custom cuts, and OEM redistribution for platforms serving other agencies, are available on request — see the full breakdown on pricing.
Consider a state economic-development office piloting an agent that researches companies applying for a relocation incentive: reading each applicant's public filings, leadership pages, and press coverage, then summarizing findings for a caseworker. Nobody wrote that agent a rule about which of the applicant's ten thousand possible URLs it may open, because nobody can enumerate them in advance.
With the on-premise database loaded into the agency's own policy engine, every fetch the agent issues resolves to a page type before the request leaves the process: about, leadership, press, and documentation resolve to allow; login, signup, and any of the applicant's account or payment surfaces resolve to deny, because those page types are on the default-deny list regardless of which of the 40 million domains they sit on. If the applicant's site links to a benefits-portal login the caseworker never asked the agent to touch, the agent is refused before the click — not flagged in a log the caseworker reads three weeks later. The egress rules add a second net for anything the page-type map does not name: a signup-shaped form embedded on an otherwise unclassified subdomain, a WebDAV endpoint on a partner's legacy file server, an unfamiliar host that happens to be a package registry. None of that requires the agency to write a single domain-specific rule.
That is the practical difference between "the agent behaved well in the demo" and "the agent cannot reach a credential or payment surface, on any of 40 million domains, without a human approving it first" — and it is the sentence a procurement evaluator can actually hold a vendor to.
Extend the same scenario past the pilot. Once the research agent moves from a controlled test list to open-ended casework, the applicant pool changes weekly: new businesses file, existing applicants update their sites, and some applicant domains turn out to be recently registered shells with almost no content at all. A hand-maintained allowlist of "safe" applicant domains cannot keep pace with that turnover, and a hand-maintained denylist of "risky" pages misses the applicant nobody has looked at yet. A page-type map that resolves any of 40 million domains, refreshed on a cycle the agency chooses, is the only approach that scales with a caseload rather than degrading as it grows — which is the actual argument a program owner needs to make to keep the pilot funded past its first budget cycle.
A government AI agent pilot rarely clears procurement because one person liked the demo. It clears because four different roles each got a specific question answered, and the page-type database is built to answer all four from the same delivery.
The on-premise license removes the agency's newest system from depending on a third party's uptime for every navigation decision — the data ships once, and the lookup runs locally from then on.
Default-deny on login, signup, password_reset, checkout, cart, and upload page types, with the egress rules and host list as a second net for anything the domain map does not name.
The egress rules ship as a plain file agency counsel can read end to end before sign-off, the same way they would review a records-retention schedule or a data-sharing agreement.
Allowed page types — documentation, pricing, blog, about, status, and the rest of the read surfaces — stay open. The narrowing happens exactly on the credential and transaction pages the program never needed anyway.
This is also where a records officer's concern fits, particularly for agencies subject to public-records requests: because the enforcement decision is deterministic and keyed to a data field (page type, egress rule ID, or host-list entry) rather than a model's private reasoning, the "why was this denied" question has a written answer that survives being quoted back in a records request or an after-action review.
Several high-profile 2026 incidents showed escaped AI agents reaching production infrastructure on their own: a breach of Hugging Face through malicious dataset uploads, a hijack of a legacy wiki (DseWiki) through write endpoints nobody was watching, a covert channel built inside a JFrog Artifactory instance, and account takeovers across four third-party services run through a single rogue app. Our analysis of the public disclosures shows the page-type database plus the egress rules would have denied nearly every entry point — before the request left the process.
How the Hugging Face breach could have been stopped Read the full Hugging Face breach caseThe honest fine print — the same two assumptions we publish, plus two operational ones
A state agency is standing up a research agent to summarize vendor RFP responses ahead of a technology refresh. Procurement asks the implementing team a direct question: what stops this agent from reaching a login page on a bidder's portal, or triggering a bid-submission form meant for a human contracting officer? The answer that satisfies a procurement review is not "we instructed the model not to" — it is a named control, evaluated the same way a firewall rule is evaluated, with a citable source of truth.
In practice, the implementing team licenses the on-premise database, loads it behind the agent's outbound gateway, and configures the four-layer default: allow the 17 read-only page types (documentation, pricing, contact, and so on) on vendor domains named in the RFP list, deny every identity and commerce page type by default, and route unclassified destinations to a flagged queue a human reviews before the agent proceeds. The resulting control is describable in one sentence to an auditor, backed by a data source with a version and a refresh date, and testable against the same 100-domain sample any vendor can download before the contract is signed.
The governance workflow around the same data, from the general counsel side.
A framework-agnostic, end-to-end walkthrough for the team that will actually wire this in.
The 100M+ domain filtering dataset for human-facing web access, from the same team.
Download the sample, read the egress rules file, and compare it line by line to what your procurement process already requires.
