A policy engine needs a policy, and a policy needs a file format someone can read, review, and diff in a pull request. This guide designs that format from first principles — why YAML, what the minimum viable schema looks like, and one complete, line-by-line annotated example covering allow, deny, and flag rules built from page types, IAB categories, and filtering conditions together.
The check function in the implementation guide hard-codes its allow and deny sets as Python literals. That's fine for a prototype. It stops being fine the moment a security or compliance reviewer, who does not read Python, needs to confirm exactly which page types an agent role can reach — and it stops entirely once you have more than one or two agent roles, each with a slightly different policy, all living as scattered constants across a codebase.
A policy file solves both problems: it separates what is allowed from how the check is implemented, so the policy can be reviewed, versioned, and diffed independently of application code. YAML is the practical choice for this over JSON for one reason that matters more than any other in a security-review context: YAML supports comments. A policy file that can explain, inline, why a rule exists is a policy file a reviewer six months from now can actually audit, rather than one that requires archaeology through commit history to understand.
There is a second, quieter benefit to treating policy as a file rather than as inline code: it becomes something a change-management process can actually govern. A pull request that changes verdict: deny to verdict: allow on the identity-group rule is a one-line diff a reviewer can catch immediately, with a required approver and a clear record of who changed it and when. The same change buried inside a two-hundred-line application file, mixed in with unrelated logic, is far easier to miss in review and far harder to attribute after the fact. Treating the policy as its own file with its own review path is a small operational decision that pays off specifically in the moment something goes wrong and someone needs to know exactly what the policy said at a given point in time.
None of this requires a heavyweight system. The schema below is deliberately small: a handful of top-level keys, a list of rules, each rule a short match block and a verdict. The goal is a format an engineer can write correctly from memory after reading this page once, not a general-purpose policy language with its own runtime and edge cases to learn. Complexity in a security control is itself a risk — a schema nobody fully understands is a schema people work around rather than extend correctly.
This is an illustrative schema for structuring your own agent web policy as a reviewable file — not a proprietary format the API requires. The API and on-premise database (see pricing and tiers) expose page types, IAB categories, and filtering categories as data; how you express your allow/deny/flag logic on top of that data is your own design decision, and the shape below is a reasonable starting point covering the conditions this site's data actually supports.
A minimal version of this schema is just two fields: default: deny and a list of page_type allow rules. That alone is a legitimate starting point and covers a large share of real deployments — a research agent that only ever needs pricing, documentation, and blog pages doesn't need category conditions or a flag verdict at all. The fuller schema below adds category conditions and the flag verdict specifically for the two situations a page-type-only policy can't express on its own: "allow this page type, but not on this kind of site" and "this action needs a human, not an automatic answer." Add those two capabilities when you actually hit a case that needs them, rather than building them into every policy file from day one.
Every line below is either a real schema field or a comment explaining it. Read it as a policy you could paste into a review, not as a code sample.
# --------------------------------------------------------------------------- # Agent web policy — illustrative schema. # Evaluated top to bottom; the first matching rule wins. No match -> default. # --------------------------------------------------------------------------- version: 1 # The fallback when nothing below matches. This is the single most important # line in the file — see the implementation guide's default-deny section. default: deny # Applies to every rule below unless a rule overrides it with its own "on". applies_to: agent_roles: ["vendor-research-agent"] rules: # --- Identity group: hard deny, no exceptions in this policy ------------- - id: deny-identity-surfaces match: page_type: [login, signup, password_reset] verdict: deny reason: "Credential surfaces — never allowed for this role" # --- Commerce group: allow read pages, deny everything transactional ----- - id: deny-transaction-surfaces match: page_type: [cart, checkout, subscribe] verdict: deny reason: "Vendor-research agent never completes a purchase" - id: allow-pricing-and-product match: page_type: [pricing, product] verdict: allow reason: "Core research targets" # --- Combined condition: category + page type together ------------------- # Deny research pages on Gambling-classified domains regardless of page type, # even though pricing/docs would otherwise be allowed above. Order matters: # this rule must appear before the general research-read rule below it. - id: deny-restricted-verticals match: web_filtering_category: ["Gambling", "Adult Content"] verdict: deny reason: "Out of scope for this agent's mandate regardless of page type" - id: allow-research-reads match: page_type: [documentation, help_center, about, status, contact, case_studies, blog] verdict: allow reason: "Standard research-read page types" # --- IAB condition: allow leadership pages, but only in one vertical ----- - id: allow-leadership-software-vertical match: page_type: [leadership] iab_v3_tier1: ["Technology & Computing"] verdict: allow reason: "Leadership research is in-scope only for software vendors" # --- Content-write group: flag, don't silently deny or allow ------------- # A rare legitimate case (e.g. an agent submitting a support ticket) should # reach a human, not fail silently and not proceed unattended either. - id: flag-content-write-surfaces match: page_type: [post_create, comment, upload, community] verdict: flag reason: "Write actions require human confirmation for this role" # --- Any-domain pattern conditions, independent of the page-type database - # These reference the bundled Egress Rules Library and High-Value Host List, egress_rules: "guardrails/egress-rules-library.php" # ~40 URL-pattern rules, any domain host_list: "guardrails/high-value-hosts.php" # ~60 curated dangerous hosts # --- What happens if the lookup itself fails -------------------------------- on_lookup_error: deny # never "allow" — see implementation guide, step 4
Two design choices worth noticing: rules are evaluated in order with first-match-wins, which is why the Gambling-category deny appears before the general research-read allow even though both could theoretically match the same URL; and every rule carries a human-readable reason string, which is what turns an audit log entry into something a reviewer can act on rather than a bare rule ID. A third, easy to miss on first read: the egress_rules and host_list references sit outside the rules list entirely, because they apply on every domain regardless of which page-type rules matched, mirroring how the underlying data actually layers.
| Field | Type | Purpose |
|---|---|---|
version | integer | Schema version, so a policy engine loading multiple files can reject or migrate an outdated shape instead of silently misreading it. |
default | allow / deny / flag | The verdict when no rule matches. This should be deny in essentially every real deployment — see the default-deny argument in the implementation guide. |
applies_to.agent_roles | list of strings | Scopes the whole file, or an individual rule via its own on block, to specific agent identities so one file can't accidentally apply somewhere it wasn't reviewed for. |
rules[].match.page_type | list of strings | Any of the 28 page types (login, checkout, pricing, documentation, and so on) from the page-types database. |
rules[].match.web_filtering_category | list of strings | Any of the 59 web-filtering categories, for vertical-level exclusions independent of page type. |
rules[].match.iab_v3_tier1 (or v2, tiers 1–4) | list of strings | IAB content-category conditions, for finer-grained vertical targeting than the filtering taxonomy alone. |
rules[].verdict | allow / deny / flag | What happens when this rule's match conditions are satisfied. flag routes to human review rather than resolving automatically. |
rules[].reason | string | Human-readable justification, carried into the audit log on every decision this rule produces. |
egress_rules / host_list | string (path or reference) | Points at the bundled Egress Rules Library and High-Value Host List, which apply on any domain regardless of page-type database coverage. |
on_lookup_error | allow / deny | The fallback specifically for a failed or timed-out lookup, kept distinct from default so an outage and an unmatched URL can be tuned independently if needed — both should normally be deny. |
A schema this small will not cover every condition a large deployment eventually needs — time-of-day restrictions, per-tenant overrides, and rate-based conditions are common additions once a policy file has been in production for a while. Add fields incrementally, and bump version whenever a new field changes how an existing file should be interpreted, rather than growing the schema silently underneath policies nobody re-reviewed against the new behavior.
Before writing a single match condition, decide what allow, deny, and flag each actually mean for your system operationally. Flag in particular needs a defined destination — a queue, a Slack channel, a ticket — or it becomes a verdict nobody ever resolves, which in practice behaves like a silent allow once the backlog is long enough that nobody reviews it promptly.
These are the rules with the least judgment call involved — see the login and checkout guides linked below — and putting them first in the file, evaluated before anything else, means a later rule can never accidentally re-open them.
Name exactly the page types the agent's role requires — not "everything except the deny list." An allow list naming five page types is easier to review than a deny list trying to anticipate every bad one.
Most rules should be page-type-only, which is easier to reason about. Reach for IAB or filtering-category conditions specifically for cases like "allow this page type, but not in this vertical," as in the schema above.
Reference the bundled any-domain layers, set default and on_lookup_error to deny, and only then run the file against the free sample CSV to confirm it produces the allow/deny split you expect.
A YAML file that looks correct and a policy engine that behaves correctly are two different things until you have actually run URLs through it and checked the results. Treat the file the same way you'd treat any other piece of logic controlling access to something sensitive.
Build a small fixture set from the free sample CSV: a handful of known login and checkout URLs that should resolve to deny under the identity and commerce rules, a handful of pricing and documentation URLs that should resolve to allow, and at least one domain from a restricted vertical (per the web_filtering_category condition in the example) that should deny even though its page type alone would otherwise allow it. Run the fixture set through your loaded policy after every change to the file, not only when the change looks related to the rules being tested — a rule reordering elsewhere in the file, or a typo in a page_type string, can silently change behavior for a rule nobody touched.
Two failure modes are worth testing for explicitly, because they are the ones a quick read-through of the file tends to miss. The first is a typo in a page-type or category string: "chekout" instead of "checkout" parses as valid YAML and matches nothing, silently falling through to whatever default is set to — deny, if you followed the guidance above, which fails safe but also fails silently unless you're specifically testing that the checkout rule actually fires. The second is rule-order drift: a broad allow rule accidentally moved above a narrower deny exception during an edit will start permitting exactly the case the exception existed to catch, and nothing about the file's syntax will flag that as wrong.
Several 2026 incidents — the Hugging Face breach via dataset uploads, the DseWiki hijack through legacy write endpoints, the JFrog Artifactory covert channel built on a plugin install and an open WebDAV path — all crossed a page type or URL pattern that a policy file structured like the one above would have denied under its default rule, without needing a bespoke rule written in advance for that specific attack. That is the actual argument for a schema with a deny default and named allow exceptions, rather than the reverse: none of these incidents required predicting the exact attack in advance, because the surfaces they crossed (upload, post/edit endpoints, plugin installs) were never named in an allow list to begin with.
The 2026 agent incidents, prevented Read the JFrog Artifactory caseThe honest fine print — the same two assumptions we publish, plus two operational ones
The architecture that loads a file like this one: cache layer, lookup, decision logic, audit log.
Why the identity-group deny rule at the top of the schema deserves that priority.
The commerce-group rules above, and the authorized-exception pattern in full.
If your policy also needs to reason about a whole class of destination independent of any specific page type — blocking agents from a category of site entirely rather than one page type on it — aitoolsblocklist.com's domain-level risk categories are designed for exactly that layer, and combine cleanly with a page-type schema like this one since they answer a different question about the same URL.
One more practical note on ownership: a policy file that lives in the same repository as the agent code it governs tends to drift toward being edited by whoever is already touching that code, for reasons unrelated to the policy itself. Keeping the file in its own reviewed location, with its own approver group, is a small structural choice that keeps the policy's history legible on its own terms rather than mixed into the noise of unrelated feature work.
Download the free sample, write the rules above against it, and confirm your allow/deny split before this touches production traffic.
