LLM data leakage: how your own apps send client data out

LLM data leakage, in the apps your own team builds, rarely looks like an attack. It is a Social Security number sitting in a debug log, a medical record stored at the provider for a month because nobody changed a default, or a retrieval step that hands the model a paragraph from the wrong client’s file. Each request works as intended, and the data still ends up somewhere you did not plan for.

Nonimo Gateway runs on your own servers and replaces client data before it leaves. See plans and get in touch.

The fix is not to stop using the API. It is to know the six routes data takes out of an LLM application and close each one, mostly by sending placeholders instead of real values. This guide walks those routes with an invented law firm, then sets out the controls in the order they pay off.

Six places LLM data leakage happens in your own code

OWASP lists sensitive information disclosure as LLM02 in its 2025 Top 10 for LLM applications, with PII named in the description. In a typical business app, it shows up in these places.

Route	How data gets out	Who ends up holding it
The prompt	Full documents pasted or attached to each request	The model provider
Application and gateway logs	Request bodies written to disk for debugging	Your servers, backups and anyone with log access
Tracing and monitoring tools	Prompts and answers sent to an observability service	Another vendor, often never reviewed
Retrieval (RAG)	Search returns passages from the wrong matter or client	Whoever asked the question, then the provider
Tool calls	The model passes personal data as arguments to an outside API	That outside service
Provider-side storage	Stored responses, uploaded files and vector stores	The model provider, until you delete them

The last row is the one most teams miss, because the defaults vary by endpoint. According to OpenAI’s data controls page, API data is not used for training by default and abuse monitoring logs are kept for up to 30 days. The Responses API stores application state for 30 days by default, while files and vector stores are kept until you delete them.

One request, traced: a personal injury firm

Pruitt & Okafor is an invented 60-person personal injury firm in Atlanta. Its developers build an assistant that drafts demand letters. A paralegal types: “Draft the demand for Marcus Bell, DOB 03/14/1981, claim 7720-44-1093, using his ER records.” Here is where that sentence goes.

1. Retrieval. The app searches the firm’s document store for “Marcus Bell ER records”. The index is shared across matters, so it also returns a page from another client’s file with a similar name.
2. The prompt. Both passages, the client’s name, date of birth and claim number go to the model API in one request.
3. The gateway log. The firm’s LLM gateway writes the full request to its database, because body logging was switched on during testing and never switched off.
4. Tracing. A monitoring tool a developer added last spring receives a copy of the prompt and the answer.
5. The tool call. The model calls a “lookup provider” function, which sends the client’s name and date of birth to an outside medical-records request service.
6. Provider storage. The app uses the Responses API with default settings, so the response object is stored at the provider for 30 days.

One question from a paralegal has put Marcus Bell’s details in six places, plus another client’s ER notes in a draft that could reach opposing counsel. Nothing was hacked. Whether redaction protects privilege is a separate issue for the lawyers, but the engineering problem is plain enough.

How to stop LLM data leakage, in order

These controls are listed in the order that removes the most risk for the least work.

1. Minimize. Send the passage the model needs, not the whole file. OWASP’s guidance on sensitive information disclosure starts from the same place, recommending least-privilege access to sensitive data.
2. Pseudonymize before the call. Replace names, dates of birth, claim numbers and account numbers with placeholders such as [PERSON_1] before the request leaves your network. The model can still draft a demand letter for [PERSON_1]. Our guide to redacting a Social Security number shows why partial masking is not enough.
3. Keep the key. The table that maps [PERSON_1] back to Marcus Bell stays on your server. Never send it to the provider, the logs or the tracing tool.
4. Restore in the answer. Put the real values back into the response on your side, so the paralegal sees the real name and nobody downstream does.
5. Filter retrieval by permission. Restrict search to the matter and the user before passages reach the prompt.
6. Turn off storage you do not use. Set provider storage options deliberately, delete uploaded files when the job is done, and keep body logging off in production.

If steps 2 to 4 happen in one place in front of everything else, the gateway log, the tracing tool, the tool call and the provider all receive placeholders. That is the argument for doing it in a proxy rather than in each app.

What US law treats as the damage

There is no single federal privacy law for this, but several rules meet at the leak. The FTC has said plainly that “there is no AI exemption from the laws on the books”, and it warns AI companies against quietly changing their terms to use customer data. In California, the Attorney General notes that consumers can sue when their “nonencrypted and nonredacted” personal information is stolen in a breach caused by poor security, so a placeholder in a log is a very different exposure from a real name. Tax and accounting firms under the FTC Safeguards Rule must notify the FTC within 30 days of discovering an event involving unencrypted information about 500 or more consumers. Our map of US state privacy laws covers the rest of the patchwork.

Nonimo Gateway

Nonimo Gateway does steps 2 to 4 for you. It replaces personal data before the request reaches the LLM API and puts the real data back in the response. It runs on your own servers, and the content does not pass through Nonimo. It is OpenAI-compatible, so you change base_url, keep your code and keep your own OpenAI key, and it has a plugin for LiteLLM.

Plans and prices are on the Nonimo Gateway page.

It helps close the routes above; your retrieval permissions, log settings and vendor agreements still need their own attention. Where it sits next to a router is covered in AI gateway for LLM apps, and whether to host a model instead in private LLM or privacy proxy. For other proxy designs, compare Kong AI Gateway and LiteLLM guardrails.

Trace one real request through your own app this week and count the places it lands. See Nonimo Gateway’s plans and get in touch to cut that count down to your own servers.

Sources

Checked October 20, 2026.

• OWASP GenAI Security Project, LLM02:2025 Sensitive Information Disclosure: PII disclosure, least-privilege access, sanitization and redaction.
• OpenAI, Data controls in the OpenAI platform: training default, abuse monitoring logs up to 30 days, Responses API storage for 30 days by default, files and vector stores kept until deleted.
• FTC, AI Companies: Uphold Your Privacy and Confidentiality Commitments, January 9, 2024.
• California Attorney General, California Consumer Privacy Act: the right to sue over nonencrypted and nonredacted personal information stolen in a breach caused by a failure to keep reasonable security.
• eCFR, 16 CFR 314.4: notice to the FTC within 30 days for notification events involving 500 or more consumers.