Every team evaluating agent guardrail data eventually asks whether to build a crawl-and-classify pipeline in-house instead of licensing one. It is a legitimate question with a real answer, and the honest answer depends on scale, timeline, and how much ongoing maintenance a team is willing to carry indefinitely — not on a single number either side can wave around.
A build decision is also a security-timeline decision. Every month spent building a crawler is a month of agent traffic running without page-type enforcement. See the incident-by-incident prevention analysis → The JFrog Artifactory covert channel is a useful case in point: catching an obscure WebDAV write path requires exactly the kind of long-tail crawl coverage that takes a from-scratch pipeline the longest to reach.
Build-vs-buy comparisons go wrong most often by comparing a license price against only the first of these four categories — usually crawl infrastructure — and ignoring the other three, which is where a self-built pipeline's real cost tends to live.
Bandwidth, proxy rotation to avoid blocking, and headless-browser rendering for JavaScript-heavy sites — a materially heavier requirement than static HTML fetching once single-page apps are in scope. Scales with domain count and re-crawl frequency, not a one-time build.
Raw fetched pages, rendered DOM snapshots, and the processed classification records themselves, multiplied across tens of millions of domains and every historical refresh you want to keep for audit or rollback purposes.
Whether classification runs on a trained model or an LLM-based pipeline, inference cost is per-page and multiplies by every page type you attempt to detect across every domain, every refresh cycle.
Crawler maintenance as sites change their anti-bot defenses, classifier retraining as page layouts drift, verification-pipeline upkeep, and incident response when the data goes stale. This category never goes to zero after launch.
Domains expire and change hands, sites redesign, and new domains appear constantly. A crawl that isn't re-run and re-verified on a fixed cadence degrades from useful to actively misleading.
Time spent building and operating a crawl pipeline is time not spent on the product the pipeline was meant to support. This category rarely shows up on a budget line, but it is real.
No single row settles the decision on its own. Read the whole table against your own scale and timeline before concluding either direction.
| Dimension | Build in-house | License (buy) |
|---|---|---|
| Time to first production use | Months of engineering across crawling, rendering, and classification before the first reliable record ships — longer if JavaScript-rendered sites and locale variants are in scope from day one. | Immediate: the sample CSV and full database are available now, with page types, IAB, and filtering categories already populated. |
| Coverage breadth | Starts narrow and grows only as fast as crawl capacity and engineering time allow; reaching tens of millions of domains is a multi-stage infrastructure effort, not a configuration change. | 40 million+ domains today, popularity-ranked, covering 99.99% of active internet usage across every tier. |
| Page-type accuracy | Depends entirely on the classifier built and how much labeled data and iteration went into it; SPA rendering, locale prefixes, and identity-provider subdomains each require dedicated engineering to handle correctly. | Every URL confirmed against the live site before being recorded — absence is recorded honestly rather than guessed. |
| Taxonomy depth | Page-type classification alone is a substantial project; adding IAB content categories (700+) and a filtering taxonomy (59 categories) on top is effectively a second and third classification system to build and maintain. | All three layers — page type, IAB category, filtering category — ship together in every record. |
| Freshness & re-verification | Requires a standing re-crawl schedule, drift detection, and ongoing engineering attention as sites change — a recurring operational cost with no natural end date. | Refresh available as an add-on at 30% of the license price per year, re-verifying URLs and screening newly registered domains on a fixed schedule. |
| Ongoing maintenance burden | Falls entirely on the team that built it, indefinitely, competing for attention with every other engineering priority. | Falls on the vendor; the licensing team consumes a CSV or API response. |
| Data control & sovereignty | Complete control over pipeline, infrastructure location, and data handling — the strongest argument for building, where it applies. | One-time database licenses run entirely on-prem with no calls back to us; the lookup API is the only mode with external dependency. |
| Upfront + ongoing cost | Engineering time across multiple disciplines (crawling, ML/classification, data operations), scaling with target coverage and refresh frequency — a genuine, non-trivial, ongoing investment rather than a one-time expense. | Fixed one-time license fee by domain tier (10M / 15M / 30M), or a monthly API plan; optional refresh at a fixed, published percentage of license price per year. |
Each of these is a real engineering project on its own, and skipping any one of them produces a pipeline that looks complete in a demo and fails in production.
Fetch pages at scale, respecting robots directives and rate limits, across a domain list wide enough to be useful — and keep doing it as domains come and go.
Execute JavaScript for single-page apps so a crawler sees the actual rendered page, not an empty shell — a meaningfully heavier compute cost than static fetching.
Determine which of dozens of page types, if any, a given URL represents, based on rendered content rather than URL string patterns alone.
Confirm a classified URL actually serves that page type on the live site before recording it as fact, and record honest absence when no such page exists.
Re-run the entire loop on a cadence that keeps pace with site redesigns, domain expirations, and newly registered domains — forever, not once.
The fifth stage is where most internal build estimates fall short, because it is the one stage that never finishes. A pipeline that runs once and produces a snapshot answers "what did the web look like on the day we crawled it" — useful for a one-time research project, but a poor foundation for a live agent-guardrail policy that has to be accurate about what the web looks like today.
A fair comparison has to look at the shape of the cost over time, not just an initial estimate, because the two paths spend very differently across a multi-year horizon.
Build: heavy upfront engineering investment across crawling, rendering, and classification, with no production-ready data until well into the year in most realistic scopes. Buy: a fixed license fee, with usable data from day one.
Build: the initial engineering investment shifts into an ongoing maintenance mode — retraining classifiers, chasing site changes, expanding coverage — that competes with new-feature work indefinitely. Buy: an optional, fixed-percentage refresh fee if currency matters; otherwise a static snapshot.
Build: the pipeline either keeps consuming maintenance attention or silently degrades as sites drift away from what the classifier was trained on. Buy: cost stays flat and predictable, tied to a published refresh percentage rather than headcount or infrastructure changes.
The build path's defining risk is not that it costs more on day one — for a narrow enough scope, it might not. It is that the cost never fully stabilizes, because the web the pipeline crawls keeps changing indefinitely, and every maintenance gap shows up later as silently stale data rather than as a visible line item.
The most expensive outcome in this comparison is not a bad build or an unnecessary purchase — it is a build that quietly stops being accurate and nobody notices until a policy engine acts on a wrong answer.
A self-built classifier's accuracy degrades gradually as sites redesign, identity providers change, and new page layouts emerge that the original training data never saw. There is rarely a clean failure signal for this kind of drift — the pipeline keeps running, keeps producing records, and the records simply become wrong at a rate nobody is actively measuring unless someone built a separate accuracy-monitoring system on top of the classification system, which is itself another maintenance burden layered on the first four cost categories.
A licensed database with a published refresh cadence makes the freshness question explicit and contractual instead of ambient: you know exactly when the data was last re-verified, because that is a stated fact about the license rather than an open question about whether anyone remembered to run the retraining job this quarter. For a security-relevant dataset specifically — one a policy engine uses to decide whether an agent may reach a given page — that explicitness is worth pricing into the comparison alongside the four cost categories above.
Unlike a build estimate, the buy side of this comparison is a fixed, published number, not a projection.
| Tier | Coverage | One-time price | Optional refresh |
|---|---|---|---|
| Database — 10M domains | Top 10M by real-world usage | $14,999 | 30% of license / year |
| Database — 15M domains | Top 15M by real-world usage | $24,999 | 30% of license / year |
| Database — 30M domains | Top 30M by real-world usage | $49,999 | 30% of license / year |
| 40M+ / custom / OEM | Full corpus or custom cuts | On request | On request |
| Lookup API (self-serve) | Full corpus, per-lookup | from $99/month | Always current, no separate refresh fee |
The honest version of this comparison has to name the cases where building is the right call, not just the cases where buying is.
If the domains that matter to your use case are a small, specific set outside general web popularity — a niche regional industry, an internal partner network — a general-purpose 40M-domain database is solving a broader problem than you have, and a targeted internal crawl of a few hundred known domains may be genuinely cheaper to build and maintain than to license at scale.
If the classification you need is specific to your domain in a way no general page-type or content taxonomy captures — a proprietary risk model tied to internal criteria — you are building a different product than a page-type database, and no license substitutes for that.
Some regulatory or contractual environments require that no third party ever crawl or hold data about certain domains, even in aggregate. That is a real constraint a licensed database cannot satisfy, regardless of on-prem delivery, because the crawl itself happened outside your organization.
A one-time academic or internal research snapshot that does not need to stay current has a fundamentally different cost profile than a live policy-enforcement feed — the freshness and ongoing-labor categories above mostly disappear if you never intend to re-run the crawl.
Outside those cases, the calculus tends to favor buying, for a structural reason rather than a marketing one: a page-type database is infrastructure that is more valuable shared across many licensees than duplicated inside each one, the same reason most companies buy their DNS resolution, their CDN, and their base map data rather than building each from scratch.
100 real domains, full schema, free — the fastest way to see what a mature crawl-and-verify pipeline actually produces.
