AI gateway for LLM apps: what it logs and when to run your own
· Written and maintained by Nonimo
An AI gateway, which many teams call an LLM gateway, is the server your applications talk to instead of calling OpenAI directly. It holds the API keys, sends each request to the right model, retries when a provider stumbles, caps spending per team and, in most setups, writes every request and response to a log.
Nonimo Gateway runs on your own servers. See plans and get in touch.
That last job is the one that deserves a second look. For a US firm whose prompts carry Social Security numbers, patient details or account numbers, the gateway log is a brand-new copy of that data, sitting in a system nobody listed when the privacy program was written. This guide explains what an AI gateway does, what its log means under HIPAA, the FTC Safeguards Rule and the CCPA, and when running your own is worth it.
What an AI gateway does, and what each job touches
Every gateway on the market does some mix of the jobs below. The right-hand column is the part vendor feature pages tend to skip.
| Job | What it does | What it sees of your client data |
|---|---|---|
| Single endpoint | Apps call one URL instead of several providers | The full prompt and the full answer |
| Key custody | Provider keys live in the gateway, not in each app | Nothing extra, but it becomes a high-value target |
| Routing and fallback | Picks a model, retries or switches providers | The full prompt, possibly sent to more than one provider |
| Caching | Reuses answers to repeated prompts | Stores prompts and answers for reuse |
| Budgets and rate limits | Caps spend per team, key or user | Usage metadata, tied to a user ID |
| Request logging | Keeps a record for debugging and audits | Often the complete request and response bodies |
Routing, keys and budgets are useful and mostly harmless. Caching and logging are where personal information starts to pile up, because both are, by design, storage.
The gateway log is a new place where client data lives
Take an invented example. Harlan & Voss CPAs, a 38-person firm in Columbus, Ohio, builds an intake assistant during tax season. Staff paste notes from client calls, the assistant drafts the engagement summary, and a self-hosted gateway sits in the middle so the IT lead can see what each team spends. To debug a formatting problem in February, he switches on full body logging. By April 15 the gateway’s database holds several thousand requests, and a good share of them include a client’s name, filing status and Social Security number.
Nobody did anything reckless. But the firm now has a table of SSNs it never meant to keep, with its own backups, its own access list and no retention rule. Three US frameworks have something to say about that table.
The FTC Safeguards Rule, for tax and accounting firms
The Safeguards Rule lists a client “obtaining tax preparation” services as one of its examples of a customer relationship, so many CPA and tax firms are inside it. The rule asks for safeguards that control the risks you identify, and it asks you to oversee service providers by “requiring your service providers by contract” to maintain safeguards. If the gateway is a hosted service, that contract has to exist. If unencrypted customer information about 500 or more consumers is acquired without authorization, the firm must notify the FTC no later than 30 days after discovery. A log table is exactly the sort of thing that turns up unencrypted in an incident.
HIPAA, for practices and their vendors
Under HIPAA, a business associate is anyone who “creates, receives, maintains, or transmits” protected health information on behalf of a covered entity. A hosted gateway that keeps prompt bodies maintains PHI, so it needs a business associate agreement like any other vendor. A self-hosted one does not need a BAA, but its log is still PHI inside your Security Rule scope. Our HIPAA guide covers which model providers sign BAAs and for which plans.
The CCPA, for everyone with California customers
The California Privacy Protection Agency reminds businesses that collection, use and retention must be “reasonably necessary and proportionate” to the purpose. A debug log nobody reads fails that test quickly. And the California Attorney General notes that consumers can sue when their “nonencrypted and nonredacted” personal information is stolen in a breach caused by a business’s failure to keep reasonable security, which makes an unredacted log the costly kind of file to lose. Our CCPA and AI guide goes further into the state’s rules.
Hosted or private LLM gateway: how to decide
A hosted gateway runs on the vendor’s infrastructure; a private one runs on yours. Neither is wrong. The question is whether you want one more outside company holding your clients’ data.
| Question | Points to a hosted gateway | Points to a private gateway |
|---|---|---|
| Does personal data pass through it? | No, or almost never | Yes, routinely |
| Who would be the service provider? | You are fine adding the vendor under contract | You would rather keep the list short |
| Who runs it? | Nobody on staff wants to | Someone can patch and monitor a container |
| Where must logs live? | Anywhere the vendor offers | On your servers, under your retention rule |
Some concrete reference points, all from the vendors’ own documentation. Cloudflare AI Gateway’s DLP can scan prompts and responses and flag or block what it finds, and when response scanning is on it buffers the full streamed response before scanning. Portkey, now part of Palo Alto Networks, offers PII redaction that replaces data with numbered placeholders, and its docs say that redaction is “irreversible by design”. LiteLLM is an open-source proxy you host yourself, and its Presidio guardrail can mask entities and, for replace operations, put the original values back in the response. Our comparisons of Cloudflare AI Gateway, Portkey and LiteLLM guardrails go through each in detail.
If your gateway handles internal tooling with no client data, a hosted service with good spend controls is a sensible choice. Once SSNs or PHI flow through, the log and the vendor contract become the main questions.
Where a privacy proxy fits in the chain
A gateway decides where a request goes. A privacy proxy decides what is in it. The two fit together, and the order matters: put the privacy proxy before anything that writes to disk.
With that order, and a proxy that recognizes those values, the Harlan & Voss log would read “client [PERSON_1], SSN [SSN_1]” instead of the real values. The provider sees placeholders too, and the staff member still gets an answer with the real name in it, because the proxy restores the values on the way back. For an app that already uses the OpenAI Python library, the change is the base URL:
from openai import OpenAI
client = OpenAI(
base_url="https://ai-proxy.harlanvoss.internal/v1", # your proxy, not api.openai.com
api_key=OPENAI_API_KEY,
)If you already run LiteLLM as your gateway, the proxy can sit inside it as a plugin instead of as a separate hop. For the broader picture of tools that find and replace personal data, see our overview of PII redaction software.
Nonimo Gateway
Nonimo Gateway is a private proxy that replaces personal data before a request reaches the LLM API and puts the real data back in the response. It runs on your own servers or in your own cloud, and the content does not pass through Nonimo. It speaks the OpenAI API, so you change base_url and keep your code, and you keep using your own OpenAI key. It has a plugin for LiteLLM.
Plans and prices are on the Nonimo Gateway page.
It helps with the problem in this guide by making sure the gateway log, the provider and its retention see placeholders. Your contracts, retention rules and HIPAA or Safeguards Rule program still do the rest. If you are weighing whether to skip the API altogether, read private LLM or privacy proxy, and for every other route data can take out of your app, LLM data leakage.
Before you switch on full body logging in any AI gateway, decide where the real names and numbers should stop. See Nonimo Gateway’s plans and get in touch to put it in front of your own gateway.
Sources
Checked October 20, 2026.
- eCFR, 16 CFR 314.2: customer relationship examples, including tax preparation; notification event; encryption.
- eCFR, 16 CFR 314.4: oversight of service providers by contract, and notice to the FTC within 30 days for events involving 500 or more consumers.
- eCFR, 45 CFR 160.103: definition of business associate.
- California Privacy Protection Agency, FAQ: purpose limitation and data minimization.
- California Attorney General, California Consumer Privacy Act: the right to sue over nonencrypted and nonredacted personal information stolen in a breach caused by a failure to keep reasonable security.
- Cloudflare, AI Gateway DLP: scanning of prompts and responses, flag and block, full buffering of streamed responses.
- Portkey, PII redaction: numbered placeholders, irreversible by design; Palo Alto Networks acquisition, closed May 29, 2026.
- LiteLLM, documentation and Presidio guardrail: self-hosted proxy, mask or block per entity, restoration of masked values.
Nonimo is the software that does this on your own computer: it redacts client names and IDs before your text reaches ChatGPT. No account needed, and the app does it without your files leaving your machine.
Common questions
What is an AI gateway?
An AI gateway, also called an LLM gateway, is a server that sits between your applications and the model APIs they call. It gives every app one endpoint, holds the provider keys, routes and retries requests, counts tokens and spend, and usually logs each request and response.
Is an AI gateway the same as an LLM gateway?
Yes, in everyday use the two names describe the same layer. Some vendors say AI gateway because they also route image, audio or embedding calls, but the job is the same: one controlled path from your code to the model providers.
Does an AI gateway store my prompts?
Most can, and many do by default, because request logs are how teams debug and track cost. Whether yours keeps full prompt bodies, just metadata, or nothing depends on its settings, so check before client data starts flowing through it.
Why does a gateway log matter under US privacy law?
Because a log full of prompts is a new database of personal information that your firm is responsible for. Under HIPAA, a party that maintains PHI for a covered entity is a business associate; under the FTC Safeguards Rule, a firm must oversee its service providers by contract; and the CCPA expects collection and retention of personal information to be reasonably necessary and proportionate to the purpose.
When should a business run a private AI gateway?
When the requests carry client data you are accountable for and you would rather not add another outside company that stores it. A hosted gateway is a reasonable choice when no personal data passes through it, or when you accept that vendor as a service provider under a written contract.
Where does a privacy proxy fit next to an AI gateway?
In front of anything that logs. A privacy proxy replaces personal data before the request leaves your network and puts the real values back in the answer, so the gateway logs, the provider and its retention see placeholders instead.