Every database license ships three things: the page-type database, the 40-rule Egress Rules Library, and the High-Value Host List. Used together, in a fixed order, they answer one question deterministically: may this agent open this URL? This page is the complete local setup: the evaluation order, a reference implementation you can lift into your proxy, the file schemas, and worked examples you can verify against the live API — which runs exactly the same order on every request.
The Full Method is one function: URL and method in, verdict out. No step is optional and no step is a fallback you add later — each catches what the previous one cannot see, and the order is what makes the result deterministic.
Is the destination host one of the ~60 entries that are sensitive by identity — cloud consoles, secrets managers, admin planes? Exact hostnames and wildcard patterns (*.console.aws.amazon.com). A hard deny here ends evaluation; a flag is remembered and carried along.
deny wins outright · flag defers to steps 2–3Does the URL exactly match one of the domain’s verified page-type URLs from the database? A match on a write surface (login, signup, checkout, upload, post_create…) is a deny; a match on a read surface (documentation, pricing, blog…) is an allow. Normalize both sides the same way: strip scheme, www., and trailing slash.
the precision layer — verified URLs, per domainDoes the URL’s path and query match any rule in the Egress Rules Library? These fire on any domain — classified or not — and are method-aware, which matters for legacy software where writes travel as GETs. This is the layer that covers the web’s long tail.
structure over identity — works on domains nobody classifiedNothing matched? A carried host-list flag surfaces now. Otherwise: read methods (GET, HEAD) allow; write methods (POST, PUT, DELETE…) deny — an agent has no business writing to a URL no layer can name.
unmatched reads pass · unmatched writes are deniedThis mirrors the evaluation the API runs, line for line. Load the three files at startup, call evaluate(url, method) from your egress hook, enforce the verdict, log the match.
Three practical notes. Subdomains: when the full host has no database row, fall back to its base domain (chat.openai.com → openai.com) — the API does the same. Performance: the database is a hash lookup and the 40 regexes evaluate in microseconds; the whole method adds no meaningful latency to an egress hook. Logging: keep the matched layer and id with every verdict — your audit trail then explains every denial by itself.
| Deliverable | Format | Fields |
|---|---|---|
| Page-type database | CSV / Parquet: domain, language, page_types | page_types is type=url;type=url;… — only verified pages appear; an absent type is a verified negative. 40M+ domains, 28 possible types per domain. |
| Egress Rules Library | page_type_rules.jsonl — one rule per line | id, page_type, group, url_regex, write_methods, default_verdict, note. 40 rules across identity, transaction, content-write, infrastructure, and admin groups. |
| High-Value Host List | high_value_hosts.csv | host_pattern (exact or *.wildcard), category, page_type, default_verdict (deny or flag), note. ~60 curated entries, reviewed by hand. |
One example rule, verbatim, so you know what to expect: {"id":"login","group":"identity","url_regex":"(^|/)(login|log-in|signin|sign-in|…)(/|\?|$)","write_methods":["POST"],"default_verdict":"deny"}
Run these through your implementation; every row is verifiable against the live API with the same URL, and your local answers should match exactly.
| URL (method) | Verdict | Deciding layer |
|---|---|---|
| https://huggingface.co/docs (GET) | allow | page_type_db · verified documentation URL |
| https://huggingface.co/new-dataset (GET) | deny | page_type_db · verified upload URL |
| https://anywiki.example/wiki.cgi?action=edit&id=X (GET) | deny | rules · wiki_edit — fires although the domain has no database row |
| https://console.aws.amazon.com/ec2 (GET) | deny | high_value_hosts · identity, ends evaluation at step 1 |
| https://smallblog.example/posts/hello (GET) | allow | default · unmatched read passes |
| https://smallblog.example/x/handler (POST) | deny | default · unmatched write is denied |
verdict and matched with your local result. A handful of golden URLs in your CI keeps the local implementation honest as you refresh data.Database licenses ship all three deliverables with quarterly or daily-refresh options — run the complete evaluation locally with zero per-request calls.
