What is an AI gateway, and when do you need a private one?

20 October 2026 · Written and maintained by Nonimo

An AI gateway, also called an LLM gateway, is a proxy between your applications and the APIs of model providers. Every app talks to one endpoint; the gateway holds the provider keys, decides which model gets the request, applies rate limits and budgets, caches repeated answers and records what went through.

Nonimo Gateway runs on your own servers. See plans and get in touch.

That last part, the record, is where privacy comes in. A gateway that logs prompts keeps a copy of every client name, email address and account number your staff or your code sent to a model. A private gateway keeps that copy on your own servers; a privacy proxy goes a step further and makes sure the personal data never reaches the log or the provider in readable form.

What an AI gateway does, and what each feature means for personal data

Most gateways, open source or hosted, bundle the same handful of features. Each one is useful, and each one touches the data in the request in a different way.

Feature	What it does	What it means for personal data
Unified endpoint	One API, usually OpenAI-compatible, in front of many providers	Every prompt from every app passes through one point, which is a good place to control it
Routing and fallback	Sends a request to another model or provider if one fails or costs too much	Data may end up at a provider you did not plan for; your contracts must cover all of them
Central keys	Apps use internal keys; the real provider keys stay in the gateway	Fewer copies of provider keys in code, and one place to revoke access
Rate limits and budgets	Caps per team, app or user	Neutral for privacy, useful for cost
Caching	Returns a stored answer to a repeated prompt	The cache holds prompts and answers, personal data included
Logging and tracing	Records requests, responses, tokens and cost	The log is a second copy of everything sent, often kept longer than the provider keeps it
Guardrails	Checks content before or after the model call	This is where personal data can be blocked, masked or replaced

The question to ask any gateway: what does it store?

A gateway that stores prompts is a database of personal data, whether or not anyone calls it that. Vendors are open about this in their own documentation. Portkey sizes its plans by recorded logs per month. AWS warns that with Bedrock Guardrails masking on, the input field in CloudWatch Logs still contains the original, unmodified request. Kong added a setting to keep sanitised items out of logs to its AI PII Sanitizer in late 2025, and LiteLLM documents a guardrail mode that runs just for logging, used to mask data before it goes to an observability tool.

None of that is a flaw. It is the reason to decide on purpose. Under the GDPR, the UK GDPR or a US state privacy law, a log of prompts about identifiable people is a record of personal data: it needs a purpose, a retention period and access controls. Four questions settle most of it:

Does the gateway log the full prompt and response, or just metadata such as model, tokens and cost?
Where are the logs and the cache stored, and in which country?
Who can read them, and for how long are they kept?
If something masks the data, does it do so before the log is written, or after?

When you need a private AI gateway

A hosted gateway is the quickest start, and for prompts with no personal data in them it is often the right call. A private one, running on your own servers or in your own cloud account, starts to pay off when one of these is true:

Prompts carry client, patient or employee data, so the log itself becomes sensitive.
A contract or professional rule limits where client data may go, and adding another processor in the middle means another review.
You want logs for cost and audit without content, kept next to your other records.
Several teams share one provider account and you need per-team keys and limits under your control.
You plan to pseudonymise, which means a table linking stand-ins to real values. That table re-identifies everyone in it, so it belongs on your side.

The fifth point is the one people miss. The UK regulator puts it plainly: pseudonymised data is personal data in the hands of someone who holds the additional information. Whoever holds the mapping holds the identities.

Where a privacy proxy fits in an AI gateway set-up

A routing gateway and a privacy proxy do different jobs, and you can have both. The order matters more than the products:

Your app sends a request to an internal endpoint.
The privacy proxy replaces names, emails, IBANs and national ID numbers with stand-ins.
The gateway routes, meters and logs the request, which now holds no readable personal data.
The provider answers using the stand-ins.
The proxy puts the real values back before the answer reaches the user.

If step 3 comes before step 2, your gateway log keeps the originals. The privacy step can also live inside the gateway: LiteLLM, for instance, supports guardrail plugins, which we compare in LiteLLM guardrails for PII. Hosted options with their own personal data features, and where they stop, are covered in Cloudflare AI Gateway and PII and the Portkey comparison.

An example: one request, two logs

Take a 30-person accountancy firm with an internal tool that drafts replies to client emails through the OpenAI API, behind a gateway that logs full requests. A member of staff asks for a reply to this (all details invented):

Draft a reply to Priya Raman (priya.raman@example.com). Her refund for account 4471-0093 was paid to IBAN GB33 BUKB 2020 1555 5555 55 on 3 October.

Without a privacy step, the gateway log and the provider both receive that sentence as written. With one in front, both receive something like this (the format is for illustration):

Draft a reply to PERSON_1 (EMAIL_1). Her refund for account ACCOUNT_1 was paid to IBAN_1 on 3 October.

The model can still write a sensible reply. The member of staff sees Priya’s name and details back in the draft, and the firm’s log records that four items were replaced without recording what they were. If the gateway log is ever exported, shared with a contractor or breached, it contains no client data.

Nonimo Gateway

Nonimo Gateway is a private proxy for that privacy step. It runs on your own servers. Its API is OpenAI-compatible, so you change base_url and keep your code. It replaces personal data before the request reaches the model and puts the real values back in the answer. It recognises the national documents and numbers of each country it serves, such as the Spanish DNI, the German Steuer-ID, the French NIR, the Italian codice fiscale and the Brazilian CPF. It also ships as a plugin for LiteLLM, so it can sit inside a gateway you already run.

You keep your own OpenAI key. Plans and prices are on the Nonimo Gateway page.

It helps with pseudonymisation; whether a given use of client data is lawful is still your call, and our guide to putting client data in ChatGPT walks through that question.

Before you choose

An AI gateway is plumbing, and good plumbing is worth having. Decide what it may store before you switch on logging, and put any pseudonymisation in front of the log. If you are weighing a self-hosted model instead, read private LLM vs a privacy proxy; for the other routes data takes out of an app, see AI data leakage. Your insurer may ask about all this too, as our cyber insurance questionnaire guide shows.

If you want your prompts logged and routed without client data in them, see Nonimo Gateway’s plans and get in touch.

Sources

GDPR, Regulation (EU) 2016/679. Article 4(5), the definition of pseudonymisation.
ICO, pseudonymisation. Pseudonymised data is personal data for whoever holds the additional information.
Portkey, pricing. Plans sized by recorded logs per month.
AWS, Bedrock Guardrails sensitive information filters. The CloudWatch input field keeps the original request.
Kong, AI PII Sanitizer changelog. The setting to skip logging sanitised items, version 3.13.
LiteLLM, guardrails quick start. Guardrail modes, including a logging mode.

All checked 20 October 2026.

This page is for information and is not legal advice. Product details of other vendors were checked on their official pages on 20 October 2026 and can change.

Nonimo is the software that does this on your own computer: it redacts client names and IDs before your text reaches ChatGPT. No account needed, and the app does it without your files leaving your machine.

Common questions

What is an AI gateway?

An AI gateway is a proxy that sits between your applications and the APIs of large language model providers. It gives every app one endpoint and usually adds routing between models, central API keys, rate limits, caching, cost tracking and logs.

Is an LLM gateway the same as an AI gateway?

Yes, in practice the two names describe the same thing. LLM gateway stresses that the traffic is calls to language models; AI gateway is the broader marketing term and some products also route image or speech models.

Does an AI gateway protect personal data?

Not by itself. A gateway's core job is routing and control, and many store the prompts they pass on, which means they also store any personal data inside them. Protection comes from a guardrail or privacy proxy that changes or blocks the data before it leaves.

When do you need a private AI gateway?

You need one when prompts carry client or patient data and you want the logs, the keys and any re-identification table to stay on servers you control. Regulated work, contracts that restrict where data goes and several teams sharing one API account are the usual triggers.

Where does a privacy proxy fit next to an AI gateway?

It sits on the path before anything logs the request. It can run as its own hop in front of the provider or as a plugin inside the gateway, as long as the personal data is replaced before it is written to a log or sent to the model.

Are prompt logs personal data?

Yes, whenever the prompts contain information about identifiable people. Under laws such as the GDPR or UK GDPR, keeping them is processing, so you need a reason to keep them, a retention period and access controls, as with any other record.