The index mixes every department into one vector store
A semantically nearby quality file shows up as “helpful context” for a process question. The vaguer the question, the easier it is to step into a neighbor document.
RAG security · knowledge-base permissions and department dataCHENGDU, CN / 2026
The model did not decide to leak.
It summarized what the retriever handed it.
A process engineer asks about a rework procedure. The knowledge-base agent answers thoroughly and includes an internal quality note the asker is not allowed to see. The file arrived as a “most relevant chunk.” RAG’s first gate is which documents may enter the candidate set. If that step only looks at similarity, the system prompt, refusal policy and output filter are all patching leaks. Yofune Labs puts knowledge-base permissions into a standardized AI agent security assessment. KUROSHIO is the validation engine that checks what retrieval hit, what the answer cited, and where that content went afterwards.
Who this page is for
- Knowledge-base and platform teams connecting multi-department manuals, policies or tickets to an internal Q&A agent
- Security and compliance teams that must prove retrieval carries identity and department labels — not only a secrecy prompt
- Business owners and CISOs who still see cross-department summaries even though there is no download button
01The problem
The problem
Permission has to happen at retrieval, not at the answer. If the model has already seen the full text, asking it to pretend it has not is unreliable. Being a logged-in employee is not the same as being allowed to see every chunk.
Retrieval uses only a similarity score
Without caller identity and without department or tenant filters, the candidate set is the whole corpus. Application login only constrains who can open the chat window, not which chunks the retriever may fetch.
“Please respect permissions” lives in the system prompt
A confidentiality clause in the prompt is not a control. If the retrieval service account can see quality files, a process question can bring those files out.
Citations cannot be opened again
If document IDs and permission labels used in the answer cannot be opened, an answer with no source cannot be production knowledge, and an overreach cannot be traced to a hit.
02The risk
The risk
Leakage does not always look like a download. A normal Q&A turn that summarizes a paragraph the asker should not see is already data overreach. Chat logs, audit trails and feedback used to improve the model may store that summary again.
Cross-department files appear as “relevant chunks”
The retriever returns slices by similarity; one of them is an internal note the asker cannot access. The agent summarizes the slices together and never hits “permission denied,” because the call itself is only retrieval.
A master service account opens the whole corpus
The dialogue layer is a directory account; the retrieval layer is a service identity that can read every document. Least privilege exists in the design doc; at runtime every question inherits the widest read scope.
Over-privileged summaries enter sessions and logs
Even without an export button, overreach is already in the answer, the session history and possibly evaluation feedback. Later spread matters as much as the original hit.
Memory and tickets turn one hit into lasting context
If a retrieval result is written into memory or a ticket summary, later questions from a different user may keep answering with that over-privileged fragment.
A passed pentest can still leave the knowledge base overreaching
A penetration test sees break-in. RAG incidents are often the wrong door: the system accepts the retrieval, but the chunk should not have entered this identity’s candidates. The two reports do not measure the same thing.
03How we test
How we test
RAG security testing looks at what retrieval hit, what the answer cited, and where that content went afterwards. Yofune Labs walks main paths with identity-bearing business questions rather than a single full-corpus scan. KUROSHIO records hits, citations and whether filtering applied.
- 01
Compare retrieval against identity, not only questions against documents
Prepare at least two department or tenant identities and ask the same class of business question. See whether candidate chunks and the final answer leave that identity’s authorization.
- 02
Inspect how the index and namespaces are cut
Whether departments share a store, whether labels apply when chunks are fetched, whether the service account can see libraries outside the task. If you cannot filter, do not connect that corpus to the agent.
- 03
Run both vague questions and named files
Vague questions jump to neighbor documents more easily. Questions that name a file confirm the filter is not limited to one phrasing.
- 04
Check citations, session logs and downstream stores
Can the answer return to a document ID and a permission label? Did an over-privileged summary land in logs, memory or feedback? An answer with no source is not production knowledge.
- 05
Retest the same path after the fix
With retrieval filters in place, the original question should not return over-privileged chunks. That path enters the regression set and runs before future index changes. KUROSHIO keeps the hit and intercept records.
05What you receive
What you receive
The conclusion handed to the owner has to be concrete: login at the chat window is not authorization at the knowledge base. Leave retrieval unfixed, and the smarter the agent, the more complete the over-privileged summary.
Knowledge-base trust-boundary diagram
Dialogue identity, retrieval identity, index or namespace, generator, logs and memory on one picture, showing where data enters and where it leaves.
Permissions versus hits
Which department corpora share an index, whether retrieval filters by identity, the service account’s real read scope, and whether cross-permission hits raise an alert.
Overreach paths
Which identity, which question, which class of chunk, what the answer cited, where it was stored afterwards. The path can be retold and regressed.
Validation evidence
The question, retrieval candidates (redacted), citations, answer digest, whether filtering applied. Someone who did not attend the test should be able to check it — not only a spoken demo.
Remediation and retest
Retrieval requests carry caller identity and filter at fetch time; split stores or namespaces by tenant and department; prompt confidentiality becomes an aid, not a control. The same path is retested.
Management summary
Which Q&A and knowledge base were in scope, whether the most important overreach is reproducible, and which retrieval controls must land before launch.
06How the assessment runs
How the assessment runs
KUROSHIO is the validation engine that keeps hit, citation and intercept records. Identity at the chat window does not replace filtering at retrieval.
- 01Map dialogue identity, retrieval identity, index splits and downstream logs
- 02Walk real business questions with at least two department or tenant identities
- 03Record hit chunks, answer citations, and residue in sessions or memory
- 04Prescribe retrieval-layer fixes, retest the same path, and write it into regression
Explore the KUROSHIO workspace
Switch between Scope, Explore, Validate and Report in an interactive workspace. See how authorized scope, task orchestration, linked evidence and a report summary fit together, using sample data to illustrate a general assessment workflow.
07Questions
Before you start.
01Why is “please respect department permissions” in the system prompt not enough?
If the model has already seen the full text, asking it to pretend it has not is unreliable. Permission has to happen at retrieval: the request carries caller identity and chunks are filtered at fetch time. If you cannot filter, do not connect that corpus to the agent.
02If there is no download button, is it still a data leak?
Yes. A normal Q&A turn that summarizes a paragraph the asker should not see is already overreach. Chat logs and feedback may store it again. The assessment looks at what was hit, what the answer cited, and where it went afterwards.
03Does RAG security testing need production corpora?
It needs corpora and identities that represent the permission boundary, not necessarily the full production store. Scope, residency and redaction are agreed before the assessment starts. Lab method explains the path; a controlled scenario is not written up as a customer win.
Fix retrieval before you launch a knowledge-base agent.
Yofune Labs’ standardized AI agent security assessment covers RAG security: knowledge-base permissions, department data overreach and citations you can open again. KUROSHIO is the validation engine that turns hits and answers into evidence. Start with the knowledge base you are connecting to an agent.