How to audit AI agent read scope across enterprise knowledge

Allowed, retrieved, disclosed, and already seen are four different sets, and each needs its own evidence.

Finding Q-1182 (illustrative audit record). The HR agent’s service account held an approved grant scoped to the HR knowledge base. For one requester, Finley Ames, the agent retrieved and cited C-4, the FY27 compensation bands file. Finley was outside the group allowed to open it.

Larkfield Systems, the 900-person software company in this record, is fictional, and so is every person, ID, date, and count below. The pattern is the point: the grant passed review, and the exposure happened anyway.

The grant says what the agent could read. It doesn’t say what the agent did read for Finley, what reached Finley’s screen, or who saw that file before.

Below is the record behind that finding: the evidence, the recheck, and the parts nobody has proven yet.

Why the approved grant didn’t answer the question

Security checked the service account first. Its approved, documented HR-only grant is a starting condition, not a finding.

An agent’s effective read scope is what it actually used and showed for a specific requester. Auditing AI agent read scope means checking four sets. Three cover one query: what the requester was allowed to read, what the retriever passed into context, and what appeared in recorded output. The fourth covers what earlier interactions already exposed.

The first three sets share one denominator, document versions. Prior exposure uses a different one, so merging any two hides the audit question.

Identities stay apart, too. A service account or an auditor may see everything, but neither view expands what the requester was authorized to read.

This audit is one narrow slice of AI knowledge management: it checks whether one agent’s behavior matched that practice’s access policy.

The evidence trail for Q-1182

Finley asked the HR agent how the promotion cycle works on September 22, 2026, at 10:14 AM UTC. Per the session record, Q-1182 ran in a fresh session with no earlier turns, attachments, or other tool calls. The retrieval trace covers the HR connector only.

Define the unit before you count. A chunk is a stored fragment of a document, and one version can produce many. Three chunks from one version count as one version retrieved.

Every row below measures Finley, Q-1182, and 10:14 AM UTC, counted in distinct document versions.

Illustrative read-scope audit – C-4 was excluded from the allowed set but appeared in retrieved and disclosed evidence; prior exposure needs a separate requester-and-date check.

Allowed comes from two sources on purpose. Many agents filter at query time against a mirrored ACL (access control list) of source permissions. That filter is only as current as the mirror, so it isn’t a live check or instant revocation. Here, the mirror and the source agreed.

They don’t always: say HR removes someone from a group at 9:00 AM and the mirror syncs at 11:00 AM. A 10:00 AM query ran on stale permissions, and rights checked at review time don’t prove rights at query time. The guide to AI access control covers authorization timing in more depth.

So the record logs source state and mirror sync time beside query time. Without both, a policy gap looks like a sync delay.

Retrieved comes from an authorized retrieval trace that lists every chunk passed to the model, mapped to its version. You have to design that trace before an incident, because you can’t reconstruct a trace that was never written. It matters even when citations look clean. Restricted text can enter context uncited, so a tidy citation list wouldn’t have proved safety either.

Disclosed comes from the recorded answer, its citations, and any text matched to a restricted chunk. Delivery uses a different unit: recorded recipients of this answer. The delivery log shows Finley only and no logged shares.

Copy-paste or screenshots wouldn’t appear in that log, so unlogged sharing stays open.

An earlier exposure

Prior exposure is separate evidence, counted as requester, version, and date, and limited to stored outputs. That search found one pair: Sam Ortiz received an answer citing C-4 on September 13. The citation disclosed C-4’s title, but no answer text matched C-4’s contents.

So the record can’t say Sam read the file’s contents. It can’t say Sam was authorized that day, either, because current membership isn’t past authority. Morgan Lee, Larkfield’s security owner, requested Sam’s September 13 group history.

The cause

A filter field in the trace shows the HR connector ran under the service account’s scope, with no requester filter. Other connectors applied one. The approved service scope had stood in for Finley’s. For the trace and metrics side of this work, see AI agent observability.

The recheck, and exactly what it passed

Dana Whitfield, Larkfield’s knowledge platform owner, enabled requester-scoped filtering on the HR connector. The connector’s change record shows the filter active from September 23 at 4:05 PM UTC.

The recheck ran Q-1182 as Finley on September 24 at 3:30 PM UTC. It used a new fresh session rather than the original chat, with the same connector and unit:

• Allowed: source and mirror both still exclude C-4 for Finley.
• Retrieved: H-7 and P-3 (2 versions from 5 chunks), C-4 absent.
• Disclosed: H-7 and P-3 cited; no output text matches C-4.
• Control: a comp-reviewers member ran the same query and C-4 was retrieved, so the filter isn’t blocking the file for everyone.

Acceptance required all four results. It passed for this requester, query, connector, and session type, and nothing wider. A recheck proves only the postcondition you measured, so the record names it in those terms.

What the fix couldn’t close

Filtering future queries doesn’t erase the matched C-4 sentence Finley received. It doesn’t remove either earlier answer from chat history. Morgan logged that exposure and notified the HR compensation owner. Notification records exposure; it doesn’t reverse it.

So the audit closed with residuals, each with an owner:

• Sam’s September 13 authorization: Morgan, pending the group history export.
• Retention for both earlier conversations: the records owner, under the existing retention policy.
• Unlogged sharing of Finley’s answer: Morgan, outside delivery-log coverage.
• The same test on the finance and legal connectors: Dana, within the week.

Turning one finding into a reusable audit record

Q-1182’s record maps onto a worksheet you can keep for the next finding. DevRev offers it as a proposed method, not an industry standard or a product output. Each row is evidence to collect, and allowed, retrieved, and disclosed never collapse into one status.

The acceptance criterion is simple to state. Every set has evidence for its stated unit and time. Every gap has a named owner plus a recheck result or a pending action.

Black-box testing alone can’t meet it. Reading an answer shows disclosure at best, never what entered context. If a platform can’t give you an authorized trace, you’re auditing outputs, not read scope.

What the record does and doesn’t prove

An audit of AI agent read scope proves boundaries for a defined test at a point in time. It doesn’t revoke access, guarantee future containment, or support a global “no leakage” claim. Each question below needs different evidence.

Q-1182’s passing recheck answers only the third row. No revocation can rewrite the fifth.

Asking a vendor for the same record

When you compare platforms, count retrieved or disclosed versions outside the allowed set, after you fix the denominator and test sample.

A benchmark won’t stand in for that count. Enterprise-Bench is DevRev’s published framework for evaluating how agents perform enterprise tasks. Treat it as task-performance context only. It doesn’t measure permissions or containment.

When a vendor shows its evidence, ask it to fill the worksheet above for one safe sample query, not a slide per feature. Two rows deserve extra pressure: how long a source change takes to reach the mirror, and which inputs the trace misses.

Then ask what the audit doesn’t prove. A credible answer names untested connectors, surfaces, time ranges, and copies. A claim that a demo proves containment or compliance is a red flag.

That test applies to Computer, by DevRev as much as any platform. Ask DevRev which of these evidence types your intended configuration can produce, and judge the answer against the same rows. Weigh what you hear against the rest of your enterprise AI agent buying criteria.

If you already hold a finding like Q-1182, or a trace you aren’t sure covers enough, review your read-scope evidence with DevRev. The conversation starts from the records you have. It isn’t a promise that any trace exists out of the box.

What’s still unproven

Finley’s answer arrived on September 22, and the recheck passed two days later. The record is useful because it says plainly what that recheck didn’t touch, and the hardest row is still open.

Then there’s Sam. A title citation on September 13 doesn’t show that Sam read C-4’s contents. Current membership doesn’t show what Sam was allowed to read that day. That row belongs to Morgan, and until the group history export arrives, its honest status is unknown.