A policy engine that allows or denies a URL is only half the system. The other half is a record good enough that, six months later, someone can reconstruct exactly why an agent was let onto a page, or stopped in front of one — without re-running the agent. This guide sets out the fields to capture per navigation decision, how long to keep them, and how to structure the log so an incident review is a query, not an archaeology project.
It is tempting to treat logging as a side effect of the policy check — write a line, move on. That produces a log that answers "did we deny it" but not "why," and the second question is the one an incident review, a compliance audit, or a confused engineer actually asks. The difference between the two is a handful of fields captured at the moment of decision, not reconstructed afterward from partial context.
Reconstruction after the fact is unreliable for a structural reason: the page-type database, the egress rules, and the host list are all versioned and can change between the moment of the decision and the moment someone investigates it. A URL that resolves to login today might have resolved to an unclassified page type when the agent actually requested it, if that domain was re-verified in between. The only trustworthy record of what the agent was told at decision time is the one written at decision time.
There is also an organizational reason this needs to be designed deliberately rather than left to whatever a logging framework does by default. The person who reviews a navigation log is rarely the person who wrote the policy check — often it is a security analyst, a compliance reviewer, or an engineer from a different team entirely, pulled in after something has already gone wrong. That person has no context on which egress rule numbers correspond to which risk categories, no memory of which page types your organization treats as sensitive, and no time to go spelunking through source code while an incident is active. The log is the interface between the policy engine and that reviewer, and like any interface, it fails silently when it assumes context the reader does not have.
These are the minimum fields for a log entry that can stand alone in an incident review, without needing to be joined against a live system that may have moved on. Add fields your organization needs (a request ID, a session ID, a cost or token count) on top of these, not instead of them.
Full URL including query string, before any redirect. If the agent's own instruction differed from the literal URL it attempted (see the injection-defense guide below), capture both.
The value returned by the lookup — one of the 28 types, or an explicit unclassified marker. Never leave this blank; a missing value is indistinguishable from a logging bug during review.
Host list, page-type database, a specific egress rule ID, or default-deny. This is what turns "denied" into "denied, and here is exactly why," which is the sentence a reviewer needs.
Record the verdict actually enforced, not just the database's raw classification — if your policy overrides a raw "allow" to a local "deny" for a specific agent role, log the enforced result.
Use the decision-time clock, in a fixed time zone (UTC), with sub-second precision if your agents can make multiple requests per second — ordering matters more than most teams expect during a chain reconstruction.
The agent instance or fleet identifier, the run or session ID, and ideally a task description or task ID. A verdict without an agent identity cannot be attributed to a deployment during a multi-agent incident.
A seventh field worth adding once the first six are solid: policy_version — the version or refresh date of the page-type database and egress rules in effect at decision time. Without it, a reviewer six months later cannot tell whether a "deny" reflects the rules as they existed then or as they exist now.
This is a conceptual sketch, not tied to any specific logging library. The structural point is that the log write happens in the same code path as the policy check, using the same lookup result — not reconstructed from a separate trace afterward.
Two structural details matter more than the exact schema: the log write happens before the caller acts on the result (so a crash immediately after cannot lose the record of what was decided), and a lookup failure or timeout is logged with its own explicit verdict — typically deny, matching default-deny — rather than silently skipped.
Retention is a balance between incident-review usefulness and storage cost, and the right answer differs by how the logs will actually get used. A reasonable default, adjustable to your own compliance obligations:
| Log tier | Suggested retention | Typical use |
|---|---|---|
| Full-fidelity per-decision log (all six-plus fields) | 30–90 days, hot storage | Active incident review, recent-behavior debugging |
| Aggregated daily summary (counts by verdict, page type, agent) | 12–24 months | Trend review, policy tuning, audit reporting |
| Denied and flagged decisions only, full fidelity | 12+ months, cold storage | Long-horizon incident review; the highest-value subset to keep longest |
| Allowed, routine reads (docs, blog, status) | Shortest retention of the four | Lowest incident value; safe to age out first under storage pressure |
The asymmetry in that table is deliberate: a denied or flagged decision is far more likely to matter in six months than an allowed read of a blog post, so if storage pressure forces a shorter retention window somewhere, shorten it on the routine-allow tier first, never on the denies.
The first question in almost every review is "what did this agent, or this fleet, request between time X and time Y." If that is not an indexed, fast query, everything downstream is slower than it needs to be.
The second most common question after an incident is "how many other agents hit this exact rule," which is how a reviewer scopes whether a single agent misbehaved or a whole fleet is walking into the same trap.
Do not hash or truncate the URL field for storage efficiency. A reviewer needs to search for a specific domain or path pattern across the full log, and a hashed field makes that impossible without a lookup table nobody remembers to maintain.
Given a run_id, the full sequence of navigation decisions for that run — in order — should be a single query. That sequence is what shows a reviewer whether an agent was steered off-task gradually or jumped straight to a denied surface.
A worked example: an agent's run_id shows twelve allowed reads of documentation and blog pages, then a denied attempt at a signup URL on an unrelated domain, then the run ends. That sequence — visible in one ordered query — tells a reviewer the agent's task drifted before the policy caught the drift, which is a very different finding from a run that goes straight from task start to a denied checkout attempt with nothing in between.
One more indexing choice pays for itself repeatedly: a secondary index on page_type alongside rule_fired. A question that comes up constantly in practice is not "what did this one agent do" but "across every agent we run, how many attempts landed on a checkout or password_reset page type last month, and were any of them allowed." Answering that from an unindexed or loosely structured log means scanning the entire table; answering it from a log designed around these fields is a filter and a count.
Abstract field lists are easier to evaluate against a concrete scenario. Here is how a well-structured log carries a security team from "something looks wrong" to a written finding, without a single re-run of the agent involved.
Say a weekly summary report — the aggregated tier from the retention table above — shows an unusual spike in denied requests from one agent fleet over a 48-hour window, most of them hitting signup and password_reset page types on domains the fleet has no history of visiting. That single aggregate number is the trigger, not the finding; the actual investigation starts by dropping into the full-fidelity log for that fleet's agent_id across the same window. Because the log is indexed by agent_id and time range first, that query returns in seconds rather than requiring a data engineer to write a one-off extraction job.
The ordered, per-run view (indexed by run_id, sorted by timestamp) shows the actual sequence: each affected run starts with several allowed reads of documentation and about pages that match the fleet's normal research task, then pivots to a burst of identity-page attempts on domains unrelated to the stated task, all denied by the same egress rule. Because rule_fired was captured at decision time rather than inferred afterward, the reviewer can see immediately that every denial in the cluster traces to one rule — the identity-surface group in the egress rules library — rather than needing to manually classify two hundred URLs by hand to notice the pattern.
The policy_version field answers the next question a reviewer always asks: was this new behavior from the agent, or a change in what the database considers a match? In this scenario the field shows no database refresh happened in the affected window, which rules out a false-positive spike from a reclassification and points the investigation back toward the fleet's own prompt or task configuration — likely a shared tool or sub-task that started generating URLs it should not have generated. None of that reasoning chain is available from a verdict-only log; it depends on having rule_fired, policy_version, and an ordered per-run view as first-class, queryable fields rather than something a reviewer would need to reconstruct from raw request traces after the fact.
The review closes with two outputs that only exist because the log made them cheap to produce: a written timeline citing exact decision_id values for the audit trail, and a count of exactly how many requests the same rule denied fleet-wide in the same window, which is what tells the team whether to treat this as an isolated task-configuration bug or a fleet-wide pattern worth a broader prompt review.
Most logging gaps are invisible until the day someone actually needs the log for an investigation, at which point they are expensive to discover. These are the ones we would flag first in a design review.
A log keyed only to the domain cannot distinguish a denied login attempt from an allowed documentation read on the same site. The page type, not the domain, is what a review needs to see.
Sampled logs work for trend dashboards and fail completely for incident review, where the one request that matters is exactly the kind a random sample is likely to drop.
"Denied" with no rule_fired value forces a reviewer to re-run the URL through the current policy to guess why — which, per the versioning point above, may give a different answer than the one the agent actually received.
When an agent acts on behalf of a specific employee or customer, that mapping needs to be queryable from the log, or every review starts with a separate request to a different team just to find out who was responsible for launching the run.
Several 2026 agent incidents ran for a long time before anyone noticed — the DseWiki hijack produced roughly 15,000 edits over about seven weeks before detection. Our analysis shows a per-decision log with the fields above would have surfaced the pattern in the first cluster of denied or flagged write attempts, not weeks later.
See the incident-by-incident prevention analysis Read the DseWiki hijack caseThe honest fine print — the same two assumptions we publish, plus two operational ones
The check function this logging pattern wraps around.
Why the URL a log records may not match what the agent's prompt intended.
What page_type values can actually appear in your log.
The four-layer enforcement model that produces every logged verdict.
The companion 100M+ domain filtering dataset, from the same team.
Download the sample, adapt the sketch above, and confirm every denied request is landing in an append-only sink.
