AI data leakage: how personal data escapes through LLM APIs

20 October 2026 · Written and maintained by Nonimo

AI data leakage from your own applications is rarely a hack. It is personal data travelling, as designed, inside API calls to a model provider: in the prompt, in the documents your retrieval step attaches, in tool results, and then into the logs your own stack and the provider keep.

Nonimo Gateway runs on your own servers. See plans and get in touch.

Stopping it does not mean stopping the calls. It means four habits: send less, replace personal data before the call, keep the key that re-identifies it on your side, and restore the real values in the answer. This guide maps where the data goes, then walks through those steps.

ChatGPT data leak or LLM data leakage?

When people search for a “ChatGPT data leak” they often mean a member of staff pasting a client file into a chat window. That is real, and we cover it in is putting client data in ChatGPT a data breach. This guide is about the quieter version: LLM data leakage from software you built or bought, which calls a model API on every request. Nobody sees it typed, it repeats thousands of times a day, and it usually carries more context than a person would paste. The OWASP project lists it as LLM02:2025 Sensitive Information Disclosure in its top risks for LLM applications, with sanitisation at the top of its prevention list.

The five routes personal data takes

Route	How it leaks	Easy to miss because
Prompt and history	The user’s text, plus the whole conversation resent on each turn	A name typed in turn one travels again in turn ten
System prompt and context	Customer records, account details or case notes added by your code	Developers add context once and forget it is there
Retrieval (RAG)	Chunks pulled from your documents into the prompt	A chunk can hold data on people unrelated to the question
Tool calls	Arguments the model sends to your tools, and the results you send back	Some protections do not look at them
Logs and retention	Your gateway, tracing tools and the provider keep copies	Logs often outlive the conversation

Two of these deserve a closer look.

Tool calls. AWS states that its Bedrock Guardrails sensitive information filters, in tool use workloads, do not evaluate tool arguments, results or definitions. LiteLLM’s Generic Guardrail API, in beta, passes tool calls to third-party guardrails, though output guardrails do not receive tool definitions. Whatever you use, check this line in its documentation.

Retention at the provider. OpenAI’s documentation says abuse monitoring logs are kept for up to 30 days by default, that API data is not used for training unless you opt in, and that the Responses API stores response data for at least 30 days by default. Zero Data Retention exists for approved customers. Those are reasonable terms, but they mean every identity you send exists in at least one more place for weeks.

And your own logs can be the larger copy. AWS notes that even with masking on, the input field in its CloudWatch logs holds the original, unmodified request. Our guide to what an AI gateway is goes through what gateways store.

How to stop AI data leakage: four steps

Minimise. Before any filter, ask what each request needs. A support summary does not need the customer’s date of birth. Trim system prompts and retrieval to the fields the task uses. Under the GDPR, Article 25, data protection by design, expects exactly this, and laws such as UK GDPR and many US state privacy laws carry similar minimisation duties.
Pseudonymise before the call. Replace names, emails, phone numbers, IBANs and national ID numbers with consistent stand-ins, so the model can still tell PERSON_1 from PERSON_2. Do it before anything logs the request, not after.
Keep the key. The table linking stand-ins to real values re-identifies everyone in it. Keep it on servers you control, for as short a time as the request needs. The ICO’s guidance is clear that pseudonymised data stays personal data for whoever holds that extra information.
Restore in the answer. Swap the real values back into the model’s response before the user sees it, so nobody has to edit placeholders by hand. If you stream answers, the restore has to work on the stream too, or users wait for the full response.

Then tidy the edges: log metadata and counts instead of full prompts where you can, set retention on traces, and review which providers your routing can reach.

An example: a support assistant with retrieval

An online retailer with 60 staff runs a support assistant on the OpenAI API, with retrieval over past tickets. An agent asks (all details invented):

Why was Mateo Rossi’s order 55-2190 delayed? Ship to 14 Harbour Lane, Leith.

Retrieval adds three past tickets. One is about Mateo; two are about other customers with similar delays, and they include a phone number and a complaint about a missed medical delivery. Without a privacy step, the provider and the trace log receive four people’s details to answer a question about one.

With minimisation, the retrieval step returns just the ticket fields the answer needs. With pseudonymisation in front of the call, what leaves looks like this (format for illustration):

Why was PERSON_1’s order 55-2190 delayed? Ship to ADDRESS_1.

The model’s answer comes back about PERSON_1, the real name is restored before the agent reads it, and the log records that a handful of items were replaced, not what they were.

Nonimo Gateway

Nonimo Gateway does steps two to four for applications that call the OpenAI API. It runs on your own servers and is OpenAI-compatible, so the change in your code is the base URL:

from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.your-company.internal/v1",
    api_key="sk-...your own OpenAI key...",
)

It replaces personal data before the request reaches the model and puts the real values back in the answer. The mapping stays on your server. It recognises the national documents and numbers of each country it serves, for example the German Steuer-ID, the French NIR and the Brazilian CPF. There is also a plugin for LiteLLM, and you keep your own OpenAI key.

Plans and prices are on the Nonimo Gateway page.

It helps with minimisation and pseudonymisation; it does not decide for you whether a given use of personal data is lawful.

Where to go next

If you are still choosing an architecture, private LLM vs a privacy proxy covers when to host your own model instead. For tools that detect personal data, see the PII redaction software overview, the Presidio alternative and Kong AI Gateway and PII. For which details count as identifying, read what to redact before you paste a document into an AI, and for the UK view, pseudonymisation under UK GDPR.

If your applications send client data to a model on every request, see Nonimo Gateway’s plans and get in touch to see what leaves once the identities are replaced.

Sources

OWASP, LLM02:2025 Sensitive Information Disclosure. The risk and its prevention headings, sanitisation included.
OpenAI, data controls in the OpenAI platform. Abuse monitoring logs up to 30 days, no training by default, Responses API data stored at least 30 days, Zero Data Retention.
GDPR, Regulation (EU) 2016/679. Article 25, data protection by design and by default.
ICO, pseudonymisation. Pseudonymised data stays personal data for whoever holds the additional information.
AWS, Bedrock Guardrails sensitive information filters. Tool calls not evaluated; CloudWatch input field keeps the original request.
LiteLLM, Generic Guardrail API. Beta; tool calls passed to guardrails; output guardrails without tool definitions.

All checked 20 October 2026.

This page is for information and is not legal advice. Product details of other vendors were checked on their official pages on 20 October 2026 and can change.

Nonimo is the software that does this on your own computer: it redacts client names and IDs before your text reaches ChatGPT. No account needed, and the app does it without your files leaving your machine.

Common questions

What is AI data leakage?

AI data leakage is personal or confidential data leaving your control through the use of AI tools. In your own applications it usually happens through ordinary API calls: data inside prompts, retrieved documents and tool calls reaches the model provider and gets copied into logs along the way.

Is a ChatGPT data leak the same as LLM data leakage?

They overlap but are not the same. A ChatGPT data leak usually means staff pasting client data into a chat app. LLM data leakage also covers what your own software sends to a model API, which nobody sees typed and which repeats on every request.

Does the OpenAI API keep my prompts?

By default, yes, for a limited time. OpenAI says abuse monitoring logs are kept for up to 30 days and that API data is not used to train its models unless you opt in. Some features, such as stored responses, keep data for their own retention period.

How do you prevent LLM data leakage?

Send less, and send stand-ins instead of identities. Minimise what each request carries, replace personal data before the call, keep the table that links stand-ins to real values on your own servers, and restore the real values in the answer.

Do RAG systems leak personal data?

They can. Retrieval pulls whole chunks of documents into the prompt, and those chunks may hold personal data about people unrelated to the question. Everything retrieved travels to the model and into any log that records the full prompt.

Are tool calls a data leakage risk?

Yes. When a model calls a tool, the arguments and the results pass back through the conversation, and some guardrail products state that they do not inspect tool calls. Check whether your protection covers them before you rely on it.