The most common way an AI system exposes sensitive information isn't through a sophisticated attack: it's an employee asking a legitimate question and receiving an answer they shouldn't be able to see. The system hasn't technically failed. It's simply done its job within the permissions it was given, which were broader than those of the person asking the question.
OWASP identifies this risk as LLM02, disclosure of sensitive information, in its Top 10 for language modeling applications by 2025. And it is the one that materializes most in business environments, precisely because it does not require anyone to act with malicious intent.
The inherited permit. The assistant queries the document repository using a service account with full access. Any user who interacts with the assistant effectively gains access to everything that account sees. This is the most common and easiest way to inadvertently gain entry.
The indiscriminate index. When building the knowledge base, an entire folder is indexed "to provide context." Inside, there might be payroll records, a confidentiality agreement, or a report on an ongoing transaction. The system doesn't discriminate: it retrieves whatever fits the query.
The exit to the outside. Someone pastes a draft contract or a piece of proprietary code into a public tool to have it improved. This is the path of shadow AI, which we have discussed in Shadow AI, the risk that comes through the corporate card.
The cumulative inference. No single answer reveals anything critical, but the combination of several allows one to reconstruct information that should not be accessible: the salary structure from fragments, a client's margin from comparisons.
The fourth one is the most difficult to control and the least anticipated, because there is no specific moment where it can be said that something was leaked.
Approach | How it behaves | Risk |
|---|---|---|
Application permissions | The system has access to everything, and the interface filters what it displays. | Any alternative route bypasses the filter. |
Permissions in the query | The documents that are retrieved are restricted depending on who is asking. | That's correct, but it depends on well-maintained metadata. |
Permissions on the data | Access is resolved at the origin, using the user's identity. | The only one robust against new forms of access |
Most enterprise systems were built with application permissions because, for twenty years, the only path to data was through the screen. An AI assistant is a new path to the same data, and that path doesn't go through the screen.
Hence the rule we apply: The AI system should consult the user's identity, not its own.. If that forces a redesign of the permissions model, it means the permissions model was already fragile and the wizard has simply made it visible.
Building a knowledge base is a security decision disguised as a technical task. Four questions before you begin:
The fourth point is often omitted due to concerns about employee privacy. This is a legitimate concern that can be addressed with a retention policy and restricted access to the record, not by deleting the record.
When hiring a tool that will read internal documentation, there are three questions that do not allow for vague answers:
Is our data used to train models? The answer should be in the contract, not on a marketing page, and it should cover both the content and the inquiries.
How are permits processed? If the answer is that the tool connects with an administrator account, there is a design problem, not a configuration problem.
Where are the embeddings and logs stored? Vectors derived from a confidential document still contain information from that document. OWASP dedicates a specific category—LLM08, weaknesses in vectors and embeddings—to this point.
There's no need to turn anything off. The order that works:
There's a test that takes five minutes and organizes the conversation better than any formal audit: Ask the assistant for something you shouldn't be able to see..
The average salary for a department. The terms of a contract with a client you don't manage. The contents of an address book. If it responds, you already know the permissions model is in the application, not the data, and you know what the first task is.
If the negation is correct, also check that it doesn't respond to the same question phrased differently. Consistency in the face of rephrasing is what distinguishes true control from a prompt instruction.
Through four avenues: inherited permissions from a service account with full access, indiscriminate indexing of folders containing sensitive documents, output of information to unauthorized external tools, and cumulative inference from several individually innocuous responses.
Use the user's account, never your own service account with broad access. If the system queries with permissions higher than the user's and the interface filters what it displays, any alternative access method will bypass that filter.
Which sources are explicitly included and which are not, whether each document has an associated access level, how content is removed from the index when its classification changes, and what is recorded from each query in order to detect and investigate a leak.
If the data and queries are used to train models, how are access permissions resolved, and where are the embeddings and logs stored? Vectors derived from a confidential document still contain information from that document.
Ask for information that the person asking shouldn't be able to see: salary details, terms of other people's contracts, or the contents of management files. If they respond, access control resides in the application, not in the data itself. It's also advisable to repeat the question in a rephrased form.
It is the reconstruction of sensitive information from several responses that, individually, reveal nothing critical. It is difficult to control because there is no specific, identifiable moment when the leak occurs, and therefore, query logging is the only means of detection.
Do you know what your AI assistant can see? We review query identity, index scope, permissions and registration, and tell you what information is exposed today. Request a review →
Español Español 
English
Français
العربية
Português
Deutsch
Italiano
