ChatGPT data leak: how your own apps expose client data

When people search for a ChatGPT data leak, they usually picture a breach at OpenAI or a staff member pasting a client file into a chatbot. For a business that has built its own tools on the OpenAI API, the more likely leak is quieter. It is your own code sending more personal information than the task needs, then copying it into logs, vector stores and tool calls that nobody reviews.

Nonimo Gateway runs on your own servers. See plans and get in touch.

Under the Privacy Act, each of those copies is personal information your business holds or has sent somewhere, and APP 11 expects reasonable steps to protect it. The fix is not to stop using the API. It is to minimise what goes out, pseudonymise what is left before the call, and restore the real values on your side.

Five ways a ChatGPT data leak starts in your own code

Path	How the data escapes	Typical example
The prompt	The app sends a whole record when a field would do	A claims summary that includes the policyholder’s date of birth and bank details
Your logs	Prompts and answers are logged in full by the app, the gateway or an observability tool	A debug log kept for a year, readable by every developer
RAG	Retrieval pulls documents that belong to someone else, or the document store sits with the provider	A search across all matters returns another client’s file note
Tool calls	Names and identifiers are passed as function arguments and results	lookup_policy(name, date_of_birth) sent back and forth with the model
Provider retention	The provider keeps logs and stored objects for a set period	Abuse monitoring logs, stored responses, uploaded files

The prompt

The OAIC’s guidance on commercially available AI products asks organisations to take steps to minimise the amount of personal information input into an AI system. In code, that usually means building the prompt from the fields the task needs instead of serialising the whole customer object. A tax file number or Medicare number should almost never be in a prompt; the identifiers an Australian document tends to hold is a useful checklist for the fields to drop.

Logs, including the ones you did not write

Logs are where most of the volume ends up. Your application may log requests for debugging, your AI gateway may log them for cost tracking, and an observability tool may keep a third copy. Guardrails do not always fix this. Amazon’s documentation for Bedrock Guardrails, for example, says the input field in CloudWatch Logs always contains the original, unmodified request, even when the sensitive information filter masks content, per its sensitive information filters page. APP 11.2 asks you to destroy or de-identify personal information you no longer need, and a log with no retention limit is the opposite.

RAG and tool calls

Retrieval-augmented generation leaks in two directions. Outward, documents you upload to the provider for search sit there: OpenAI’s data controls page says files are kept until deleted and that vector stores are retained indefinitely if you never delete them. Sideways, a retrieval step that is not filtered by client can put one client’s file in front of another, which is an unauthorised disclosure in plain terms.

Tool calls carry the same risk with less visibility. Function arguments and results travel to the model as part of the conversation. The same Bedrock page notes that in tool use workloads, the guardrail does not evaluate tool definitions, arguments or results, so any masking applied to the message text does not reach them.

Retention at the provider

OpenAI says API data is not used to train its models by default. It also says abuse monitoring logs are retained for up to 30 days, stored responses for 30 days by default, and that zero data retention is available to approved customers. That is a reasonable set of commitments, and it is still a copy of your clients’ details outside your network for up to a month. More on what each OpenAI product keeps is in does ChatGPT save your data.

What the Privacy Act makes of it

Sending personal information to an AI provider is not, by itself, a data breach. It is a use or disclosure you need to justify under APP 6, and for an overseas provider APP 8 applies, with section 16C making you accountable for what the recipient does. The OAIC’s APP 11 guidance expects your security steps to cover third party and cloud providers.

A leak becomes a Notifiable Data Breaches question when there is unauthorised access to or disclosure of personal information, serious harm is likely, and you cannot prevent that harm with remedial action. An exposed log bucket or a RAG answer that shows client B’s file to client A both fit the description. Once you suspect one, you have 30 days to assess it. If your cyber policy is meant to respond, check what an Australian cyber policy covers before you need it.

How to stop it: minimise, pseudonymise, keep the key, restore

1. Minimise in your code. Build prompts from fields, not records. Drop TFNs, Medicare numbers, bank details and dates of birth unless the task truly needs them.
2. Pseudonymise before the call. Replace the names, addresses, emails and phone numbers that remain with consistent placeholders, so the model can still tell people apart.
3. Keep the key on your side. The table that maps placeholders back to real values stays on your servers, and so does your API key.
4. Restore in the answer. Put the real values back into the model’s reply before your user sees it, including when the reply streams.
5. Log the counts, not the content. Record how many items were replaced and of what type, and keep prompt text out of long-lived logs.

A worked example: a claims assistant at an Adelaide broker

An invented insurance broker, Torrens Risk Partners, gives its claims team an assistant that answers questions about a client’s policy. It uses RAG over policy documents and a tool that looks up the policy record.

A staff member types: “Mei Lin Tran, born 14 March 1988, says water damage at 7 Kauri Court, Mawson Lakes isn’t covered. Is she right?”

Copy	Before	After minimising and pseudonymising
Prompt to the model	Full name, date of birth, street address	Client A, Address A; date of birth dropped
Tool call arguments	lookup_policy("Mei Lin Tran", "1988-03-14")	The lookup runs on your side by internal ID; the model sees Client A
RAG results	Searched across every client’s documents	Filtered to this client’s policy wording
App log	Full prompt and answer, kept indefinitely	Counts of replaced items, 30 day retention
Answer to staff	Correct	Correct, with “Mei Lin Tran” restored

The model still reads the policy wording and the question, which is what it needs to answer. It never receives who the claimant is or where she lives.

Nonimo Gateway, in short

Nonimo Gateway does steps 2 to 4 as a private proxy. It replaces personal data before the request reaches the model API and restores the real data in the response. It runs on your own servers, and it accepts OpenAI-style requests, so the change in your code is the base URL:

from openai import OpenAI

client = OpenAI(base_url="https://ai-proxy.yourfirm.internal/v1")

It runs as a LiteLLM plugin if you already use that gateway, and you keep your own OpenAI key. Plans and prices are on the Nonimo Gateway page. Step 1 stays in your code, where it belongs. For other approaches, see PII redaction software for LLM apps and LiteLLM PII guardrails. For the traffic layer, read what an AI gateway is, and for the self-hosting question, private LLM or privacy proxy.

Close the gaps before someone else finds them

Trace one request through your app and write down every place a copy lands: the prompt, each log, the document store, the tool calls, the provider. Then cut each copy down to what the task needs. A pseudonymising proxy in front of the API covers the copies your code cannot easily remove. See Nonimo Gateway’s plans and get in touch.

Sources

Checked on 21 October 2026 unless stated.

• OAIC, Guidance on privacy and the use of commercially available AI products, published 21 October 2024, updated 17 January 2025. Minimising personal information input, providers collecting inputs for training, APP 8.
• OAIC, APP Guidelines chapter 11, updated 3 October 2025. APP 11.2 destruction or de-identification, third party and cloud providers.
• OAIC, APP Guidelines chapter 8, updated 3 October 2025. Overseas recipients and section 16C.
• OAIC, Data breach preparation and response, Part 4. The three criteria of an eligible data breach and the 30 day assessment in section 26WH(2).
• OpenAI, data controls in the API platform. Training default, abuse monitoring logs, application state retention, zero data retention.
• Amazon Bedrock Guardrails, sensitive information filters, checked 20 October 2026. CloudWatch invocation logs keep the original input; tool use not evaluated.

The broker, its staff and its client are invented, and so are the address and date of birth.