robots.txt, bot detection, forward proxies and prompt guardrails all get proposed as the reason you do not need page-type data. Usually they are solving an adjacent problem in the opposite direction. These eight pages say which, plainly, including the two cases where you genuinely do not need us.
The fastest way to place a technology is to ask what question it is built to answer. Where the question differs from ours, both belong in the stack. Where it is the same question, one of them is going to be cheaper or more accurate.
| The alternative | The question it answers | Where it stops | Verdict |
|---|---|---|---|
| robots.txt | What does this site want crawlers to index? | It is the site's voice, advisory, and says nothing about what your agent is permitted to do. | Both |
| Bot detection | Is this inbound visitor to my site a bot? | Points inbound. Agent governance points outbound at your own agents. Different buyer entirely. | Opposite direction |
| Forward proxy filtering | Is this domain in a category we allow? | Domain and category granularity. It cannot tell the pricing page from the checkout page on one host. | Both |
| Prompt guardrails | Is the model being asked or led to misbehave? | Operates at the model layer, so it fails exactly when an injection succeeds there. | Both |
| Web categorization | What kind of site is this? | Answers what, not where. Combined with page types the policy gets sharper than either alone. | Both |
| Framework guardrail hooks | Where can I intercept my agent's actions? | Gives you the hook point but no data to decide with. It is the socket, not the plug. | Both |
| A hand-curated list | Which of our 50 known vendors is safe? | Fine at 50 domains. The drift math breaks it well before you reach a few thousand. | Depends on scale |
| Crawling it yourself | Can we produce this data in-house? | Yes — the question is crawl infra, classification, freshness and labor, costed honestly. | Depends on scale |
Each page works the argument properly, including the conditions under which the alternative is the right call.
An honest TCO comparison across crawl infrastructure, classification compute, freshness cycles and the labor to keep it all running.
Manual vs licensedA 50-domain list maintained by hand decays within weeks. The drift math worked in detail against a continuously re-verified database.
robots.txt vs policyrobots.txt is the site's voice telling crawlers what to index. An allowlist is the operator's policy for its own agents. You need both.
Inbound vs outboundSame word, opposite direction. One protects your site from arriving bots; the other controls what your agents do out on the web.
Proxy vs page typeForward proxies filter at domain and category level. Exactly where that granularity runs out and page-type data has to begin.
Model vs networkEach catches a class of failure the other misses. Neither alone covers the gap, and the overlap is smaller than it looks.
What vs whereA category taxonomy says what kind of site this is. Page types say where a function lives on it. The combined schema beats both.
Hook pointsA neutral map of where each agent framework exposes a hook: tool wrapper, guardrail hook, proxy or browser extension.
If your agents touch a fixed set of a few dozen internal domains, a hand-written list is cheaper and you should keep it — the comparison page says so explicitly. If your problem is inbound bots hitting your own site, bot detection is the product and this is not it. And if you have the crawl infrastructure already running for other reasons, the build-vs-buy page gives you the numbers to decide rather than an argument for our side of it.
A run of high-profile 2026 incidents saw AI agents escape their task or get steered off it — the Hugging Face breach, the DseWiki hijack, the JFrog Artifactory covert channel, a series of third-party account takeovers. Our analysis shows the documented attack steps in almost all of them would have been denied at the request. It is a useful test to run against whichever alternative you are weighing.
Would your agents have been stopped? Check the incident analysisThe honest way to settle a build-vs-buy argument is to test the data. The free sample CSV has 100 real domains in the production schema, the evaluation guide explains how to spot-check it, and pricing has the tiers to compare against your own build estimate. On the categorization side of the comparison, websitecategorizationapi.com is the sibling product.
If page-type data made the cut, these are the next three questions: how it is built, how it integrates, and what your sector requires.
Schema, coverage tiers, and the detection methodology behind login and checkout URLs.
Guides — 15Policy design, framework hooks, engine architecture and audit logging.
Industries — 12Regulated buyers, and the vendors who OEM the data into their own products.
