The lookup API and the on-premise page-types database are both raw material, not a finished policy engine. If you're building the layer that sits between your agents and the network — the thing that actually decides allow, deny, or flag for every outbound request — you need a cache strategy, a lookup path, decision logic that combines multiple conditions, an audit log, and a latency budget the whole thing has to fit inside. This guide is that architecture, end to end.
Teams that set out to "add guardrails" to an agent platform often end up building one large, tangled piece of middleware that fetches data, applies rules, and logs results all in the same function. That works until it needs to change, at which point every part is entangled with every other part. Split it into four components with a narrow, well-defined interface between each, and the whole thing stays maintainable as your policy grows.
This is also the point where it's worth being honest about whether you should be building this at all versus buying an existing AI gateway, enterprise browser, or CASB product that already ships an enforcement point. Building your own policy engine makes sense when you need it embedded directly in your own agent runtime, when an existing gateway doesn't yet support page-type-level conditions rather than only domain or category rules, or when your deployment model rules out an external product in the request path. If none of those apply, evaluate whether an existing product with page-type support already covers your case before committing engineering time to the architecture below — the four components here are the same ones any capable off-the-shelf enforcement point already has to implement internally.
An in-process or shared cache in front of the lookup, keyed on the normalized URL, with a TTL shorter than your data refresh cycle.
The API call or local database read that resolves a URL to a page type, IAB category, and filtering category on a cache miss.
Your policy file (see the YAML schema guide) evaluated against the resolved record to produce allow, deny, or flag.
An append-only record of every decision: URL, resolved fields, matching rule, verdict, and timestamp.
Page types change slowly relative to agent request volume: a domain's login URL doesn't move day to day, so caching aggressively is not a risky shortcut, it's the correct design.
Key the cache on the exact normalized URL for the url= mode (scheme, host, path, and any query parameters that affect the resolved page type) and separately on the bare domain for cases where you only need the page-type map. Set the TTL below your data's refresh cadence — a quarterly-refresh license or a monthly API cycle both comfortably support a cache TTL measured in hours to a day, which is enough to absorb the overwhelming majority of repeat traffic to the same domains any given agent fleet actually visits. A shared cache (Redis, Memcached, or an equivalent) in front of a fleet of agents multiplies the benefit further, since one agent's earlier lookup of stripe.com/login serves every other agent's subsequent request for the same URL without a second network round trip.
Cache the resolved record, never the final allow/deny/flag verdict. If you cache the verdict itself, a policy file change (say, tightening a rule) won't take effect for any URL until its cache entry naturally expires. Caching the underlying page-type and category data, and re-running the decision layer fresh on every request, keeps policy changes effective immediately while still avoiding a lookup on every cache-hit request.
The lookup's job is narrow: given a URL, return what's known about it. The decision layer's job is separate: given what's known, and the policy file for this agent role, return a verdict. Collapsing these into one function is the most common design mistake in a homegrown policy engine, because it makes the policy impossible to test independently of the network call that feeds it.
Notice that decide() takes no network dependency at all — it's a pure function of a record and a role, which means your policy logic can be unit-tested against hundreds of synthetic records in milliseconds, with no lookup client, cache, or network involved. This separation is also what makes the evaluation procedure in the data-evaluation guide practical: you can run your real decision logic against the free sample CSV directly, without standing up the rest of the engine first.
An audit log exists for two moments that matter far more than everyday operation: the incident review after something went wrong, and the compliance question about what an agent could reach on a given date.
Log the exact URL requested, the resolved page type and categories (or the explicit absence, for an unclassified domain), which rule or default fired, the verdict, the agent identity and role, and a timestamp. Treat these fields as the minimum, not a ceiling — a redirect chain worth recording as a sequence rather than a single final URL, or a request ID that ties a navigation decision back to the specific agent task that triggered it, both make a later review meaningfully faster without adding real cost to the write path. Retention should outlast your typical incident-discovery window — DseWiki ran roughly seven weeks before disclosure, which is a useful reference point for how long "we'll notice eventually" can actually take in practice. Write the log entry after the decision on every single request, allow included, not only on denies: a log that only records denials cannot answer "what did this agent access on March 3rd," which is frequently the exact question a post-incident review needs answered first.
A policy engine that adds noticeable latency to every agent navigation will get bypassed under deadline pressure, quietly or otherwise. Budget for it explicitly rather than discovering the number in production.
| Stage | Cache hit | Cache miss (API) | Cache miss (local DB) |
|---|---|---|---|
| Cache lookup | <1ms | <1ms | <1ms |
| Resolution (network call or local read) | — | 20–150ms, network-dependent | <5ms, local index read |
| Decision (rule evaluation) | <1ms | <1ms | <1ms |
| Audit log write | <1ms, async | <1ms, async | <1ms, async |
| Typical added latency | <5ms | 20–150ms | <10ms |
The practical implication: a high-cache-hit-rate deployment on the API barely notices the policy engine exists, while a cold-cache burst against many distinct, previously unseen domains will feel the network round trip on every one of them. If your workload is dominated by a long tail of one-off domains rather than repeat traffic to a smaller set, the on-premise database's local-read path removes the variable entirely, at the cost of the licensing and refresh-cycle tradeoffs covered in the evaluation guide. Write the audit log asynchronously in either case — there is no reason a logging write should sit in the critical path of a navigation decision the agent is waiting on.
Measure this budget against your actual traffic shape before committing to a delivery mode, not against the worst case in the abstract. An agent fleet that mostly re-visits a stable set of vendor domains — the common case for procurement, sales-intelligence, and support agents covered elsewhere on this site — will see cache-hit rates high enough that the API's network cost barely registers in aggregate. A fleet doing broad, exploratory crawling across a constantly shifting set of unfamiliar domains is the profile that benefits most from the local-database path, precisely because it structurally can't build up a warm cache the way repeat-domain traffic does.
Most deployments don't stay at one agent with one policy for long. The architecture above scales to many roles without changing its shape, provided you keep two things separate from the start.
First, keep the cache and lookup components shared across every agent role — there's no reason a procurement agent's lookup of a vendor's pricing page and a sales-intelligence agent's lookup of the same URL should populate separate caches or trigger separate API calls. The resolved record is a fact about the URL, independent of which agent asked. Second, keep the decision component parameterized by agent role from day one, even if you currently only have one role — retrofitting role-awareness into a decision layer that assumed a single global policy is a much larger change than building it in from the start, and the policy-file guide's per-role file pattern is designed to slot directly into the decide(record, agent_role) interface shown above.
As the number of roles grows, the audit log becomes the place most teams first feel real operational pressure: a shared log across dozens of agent roles needs to support filtering by role, by verdict, and by time range well before it needs any other feature, since "what did agent role X access last week" is the question a reviewer actually asks. Design the log's queryability around that question early, rather than treating it as a plain unstructured stream to be figured out later.
A policy engine's failure behavior is as much a design decision as its happy path. Three specific failures deserve an explicit, tested answer rather than whatever the code happens to do by accident.
The lookup times out or errors. Resolve to an unclassified record, and let the decision layer's default-deny handle it — never let a lookup exception propagate as an unhandled error that crashes the request path, and never catch it in a way that silently falls through to allow.
The cache is unavailable. Degrade to calling the lookup directly on every request rather than failing the policy check entirely. Slower is an acceptable degraded mode; skipping the check is not.
The policy file fails to load or parse. This should be a hard startup failure, not a runtime fallback to "allow everything," and it should alert loudly. A policy engine that silently runs with no rules loaded is functionally indistinguishable from having no policy engine at all, which is a considerably worse failure than refusing to start.
All three failure modes share the same underlying principle laid out in the four-layer enforcement model: every layer, including the ones you build yourself on top of the licensed data, should fail toward deny, never toward allow. A policy engine is, in effect, a fifth layer sitting on top of the database, the egress rules, and the host list — and it inherits the same obligation those layers already carry to fail closed rather than open.
The 2026 incidents share a common thread: none of them needed the target's defenses to be perfect, only for one enforcement point to fail open once. A policy engine built with the failure modes above left unhandled — a timeout that silently allows, a policy file that fails to load and defaults to permissive — is exactly the kind of single point of failure the Hugging Face breach, the DseWiki hijack, and the JFrog Artifactory covert channel exploited in their respective target systems. Our analysis shows a correctly fail-closed engine, evaluated against the database and egress rules, would have denied every documented entry point pre-request.
Every 2026 agent escape, mapped to the rule that stops it Read the Hugging Face breach analysisThe honest fine print — the same two assumptions we publish, plus two operational ones
The rules the decision component in this guide loads and evaluates.
Test the resolution component's data source before it's load-bearing.
The specific hard-deny rule your engine's decision layer should never fail to enforce.
A policy engine designed to reason about page types combines naturally with a domain-level risk feed for agents that also need to reason about entire classes of destination — aitoolsblocklist.com's categories are a common second input to the same decision layer, resolved and cached the same way as the page-type record above.
Download the free sample, wire up the four-component sketch above, and confirm your policy resolves correctly before this touches production traffic.
