Private LLM or privacy proxy: what a US firm really needs

You probably do not need to host your own model. A private LLM, meaning a model running on hardware or a cloud account you control, makes sense in a few specific situations: a contract that bans any outside processing, a site with no internet, or a plan to train on your own records. For most US firms the worry behind the question is narrower: client names and numbers reaching a model provider.

Nonimo Gateway runs on your own servers, so you keep the API and keep client data in-house. See plans and get in touch.

That narrower worry has a narrower fix. A privacy proxy on your own network swaps personal data for placeholders before the request leaves, and puts the real values back in the answer. You keep the hosted model, and the provider never sees who the request was about. This guide compares the two routes honestly, including what an on-premise LLM costs you in hardware and people.

Three things people mean by private LLM

The phrase gets used for setups with very different privacy properties. Sort out which one is on the table before comparing prices.

What is meant	Where the model runs	Who sees your prompts
On-premise LLM	Servers in your office or data center	Your own staff and systems
Model in your cloud account	GPU instances you rent from a cloud provider	Your staff, under your cloud contract
Business plan of a public AI service	The provider’s infrastructure	The provider, under its business terms

The third row is often sold as private, and it may be a perfectly good option. It is still someone else’s server, and the protection comes from the contract and settings. On the OpenAI API, for example, OpenAI says data sent to it is not used to train its models unless you opt in, keeps abuse monitoring logs for up to 30 days by default, and offers zero data retention subject to its prior approval, per its data controls page.

What running an on-premise LLM actually takes

Vendors of GPU servers are happy to quote the box. The box is the smaller part of the cost.

• Hardware sized to the model. Larger open-weight models need more GPU memory, and the model you can afford to host may not be the one your staff are used to. Plan for spare capacity, because a single server is a single point of failure.
• People. Someone has to deploy the model, patch the operating system and drivers, watch for outages, and pick up the phone when it stops answering at 4:45 p.m. on the last day of the month. In a 30-person firm that person usually already has a full-time job.
• Quality checks. A self-hosted model will not produce the same answers as the hosted one. Somebody has to test it on real tasks, such as appeal letters or engagement summaries, and decide whether it is good enough.
• Its own data footprint. The model server keeps logs, caches and backups. Hosting it in-house does not remove sensitive data from the picture; it moves it to a system you now have to secure, audit and include in your risk analysis.

None of this makes a private LLM a bad idea. It makes it an infrastructure project rather than a privacy setting.

When a private LLM is the right call

• A client contract, government agreement or regulator forbids any third-party processing of the data, with no room for a vendor agreement.
• The system has to run with no internet connection at all.
• You plan to fine-tune a model on your own records. That raises bigger questions about the training data, but it does require a model you control.
• Your request volume is steady and large, and you have the people to run it.

Keeping the API: when a privacy proxy is enough

For most practices and professional firms, none of the four apply. What they need is for the provider to receive as little as possible, and HIPAA already points that way. Its minimum necessary standard asks covered entities and business associates to make “reasonable efforts to limit protected health information to the minimum necessary” for the purpose. A model drafting a letter does not need the patient’s real name to draft it well.

An example from a dental group

Take an invented practice: Juniper Family Dental, four offices around Raleigh, North Carolina, with a front-office team that drafts insurance appeal letters. The practice manager gets a quote for an on-premise LLM and a second option from IT: keep the OpenAI API, add a privacy proxy, and sign OpenAI’s business associate agreement for the API.

	On-premise LLM	API plus privacy proxy
Model quality	Whatever the practice can host	The hosted model staff already use
Who runs it	The practice, or a contractor	A proxy on an existing server
What the provider receives	Nothing	Placeholders such as [PATIENT_1]
Vendor agreement	None for the model	BAA with the API provider still advised
Where real names live	Model server, its logs and backups	The practice’s own systems

Juniper chooses the second column. The letter goes out as “Appeal for [PATIENT_1], date of service [DATE_1]”, the model drafts it, and the proxy restores the real name before the staff member sees it. Our HIPAA guide explains which OpenAI API offerings come with a BAA.

Pseudonymized is not the same as de-identified

Be precise about what the proxy achieves. HIPAA de-identification has its own tests: the safe harbor list of 18 identifiers, or an expert determination that the risk of identifying someone is very small. Replacing names and dates in a prompt reduces what leaves the practice, but it does not automatically turn PHI into de-identified data. HIPAA does allow a covered entity to keep a re-identification code, as long as the code is not derived from information about the person and the mechanism is not disclosed. A proxy that keeps the mapping on your server follows the same idea, but keep the BAA anyway. Our guide to de-identifying data before AI covers the two HIPAA methods and the CCPA’s own definition.

Nonimo Gateway

Nonimo Gateway is the privacy proxy route. It replaces personal data before the request reaches the LLM API and puts the real data back in the response. It runs on your own servers or in your own cloud, and the content does not pass through Nonimo. It is OpenAI-compatible, so you change base_url and keep your code and your own OpenAI key, and it has a plugin for LiteLLM.

Plans and prices are on the Nonimo Gateway page.

If the proxy will sit next to a router, read what an AI gateway logs. If you want the full list of ways data escapes from your own code, see LLM data leakage. To compare open-source options, our pages on Microsoft Presidio and PII redaction software lay out the field.

Host your own model when you have a reason no contract can solve. For everything else, keep the API and keep the names. See Nonimo Gateway’s plans and get in touch to use it with your own OpenAI key.

Sources

Checked October 20, 2026.

• eCFR, 45 CFR 160.103: definition of business associate, including those who maintain or transmit PHI.
• eCFR, 45 CFR 164.502: the minimum necessary standard in paragraph (b).
• eCFR, 45 CFR 164.514: de-identification by expert determination or safe harbor, and the re-identification code in paragraph (c).
• OpenAI, Data controls in the OpenAI platform: no training on API data by default, abuse monitoring logs kept up to 30 days, zero data retention subject to approval, and BAA-eligible endpoints for PHI under an executed Business Associate Addendum.