How to deploy an AI gateway on your own server with Docker

A self-hosted privacy proxy for LLM APIs is one more container on a server you already run. Personal data in each prompt becomes a placeholder before the provider sees it, and the reply comes back with the real values. New to the idea? Read our plain explanation of what an AI gateway does first.

This is the hands-on part, from an empty Linux host to HTTPS. The commands use Nonimo Gateway, a private container image; the Compose and nginx settings suit most proxies. Still undecided? I suspect a Compose file won’t settle that; first weigh running your own model against a proxy in front of someone else’s.

What the host needs before you pull the image

A Linux VM or server on amd64 or arm64, inside your network or cloud account, with Docker Engine and the Compose plugin. If docker compose version answers, you have both. The gateway reads and rewrites requests; it does not generate text, so the model’s hardware stays your provider’s problem.

The image is private. You choose a plan on the Nonimo Gateway page, and the licence email brings registry credentials and install notes. Those notes give the exact image reference and the setting names the container reads; this guide uses placeholders wherever they are the source of truth.

Keep the network rules short: inbound only from the services that call a model, outbound only to the places the table lists. A gateway the whole office LAN can reach is asking for trouble, as I see it: sooner or later someone points a browser at it.

docker login <registry-from-your-licence-email>

Writing the Docker Compose file: three files, one folder

Make a folder such as /srv/llm-gateway and put three files in it. .env is read by Compose itself and pins the image to a version tag. gateway.env is read by the container: licence, upstream provider and, if you enable it, the access token your services must present.

# .env
GATEWAY_IMAGE=<image-reference-from-the-install-notes>:<version>

On anything that sits in the path of client data, my preference is a pinned tag over latest. It turns every update into a decision someone made on purpose, and it gives you a line to put back when you roll back. Lock gateway.env with chmod 600 gateway.env; it holds secrets.

# compose.yaml
services:
  gateway:
    image: ${GATEWAY_IMAGE}
    restart: unless-stopped
    env_file: gateway.env
    ports:
      - "127.0.0.1:8080:8080"
    logging:
      driver: json-file
      options:
        max-size: "20m"
        max-file: "5"

Storage is left open on purpose. The audit log and your client list live on your server, and the install notes say whether that path needs a volume to survive a new container. An audit trail lost on update won’t help when an insurer’s questionnaire asks how you control AI use.

First start: docker compose up and a test request

Start it and read what it says about itself:

cd /srv/llm-gateway
docker compose up -d
docker compose ps
docker compose logs --tail 50 gateway

A container marked as running is only half the answer. The other half is a real completion through the whole chain: container, licence, outbound network and provider key. Export the key as OPENAI_API_KEY and call the loopback address, adding the access token header if you enabled one.

curl -sS http://127.0.0.1:8080/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-5.5", "messages": [{"role": "user", "content": "Answer with one word: ok"}]}'

I’d demand that one-word reply before pointing a single service at the gateway. If you want to see what a raw request carries before it is masked, the fields in an OpenAI API call where personal data sits are set out separately.

Pointing the OpenAI and Anthropic SDKs at the proxy

Your services keep their SDK and their key; only the address changes. Both official Python SDKs take it in the constructor or from an environment variable, and the variable is tidier because no code moves. Note the difference in the table: OpenAI’s address ends in /v1, Anthropic’s does not.

from openai import OpenAI

client = OpenAI(base_url="https://llm-gateway.example.internal/v1")
reply = client.chat.completions.create(
    model="gpt-5.5",
    messages=[{"role": "user", "content": "Write a two-line reminder to Thea Mountjoy about Friday's inspection."}],
)

The Gateway speaks Chat Completions and Responses, with function calling and streaming, plus Claude’s Messages API. Each instance points at one provider, so OpenAI plus Anthropic means two instances. Already routing through LiteLLM? See the two ways Nonimo plugs into a LiteLLM proxy first.

Services can keep sending their own provider key in the Authorization header, or the Gateway can read it from a secret or an environment variable. For a new deployment, my preference is the second: one place to rotate the key, nothing for a developer to paste into a test script.

TLS with an nginx reverse proxy, without breaking streaming

Prompts carry exactly the data you are protecting, so they should not cross your network in plain HTTP. Add nginx to the same Compose file, under services:, terminating TLS with a certificate your clients already trust:

  proxy:
    image: nginx:1.30-alpine
    restart: unless-stopped
    depends_on: [gateway]
    ports:
      - "443:443"
    volumes:
      - ./nginx/gateway.conf:/etc/nginx/conf.d/default.conf:ro
      - ./certs:/etc/nginx/certs:ro

server {
    listen 443 ssl;
    server_name llm-gateway.example.internal;
    ssl_certificate     /etc/nginx/certs/llm-gateway.crt;
    ssl_certificate_key /etc/nginx/certs/llm-gateway.key;
    client_max_body_size 20m;
    location / {
        proxy_pass http://gateway:8080;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_buffering off;
        proxy_read_timeout 300s;
    }
}

nginx’s defaults suit web pages, not token streams, and three of them bite here. Buffering is the one I dislike most, because nothing errors: the chat simply looks broken.

Docker adds its own catch: published ports are routed before ufw rules apply. On a host with a public interface, publish on the internal address only, such as "10.20.0.15:443:443", and close 443 at the security group. Weighing Kong’s sanitiser instead? Its PII service raises its own TLS and memory questions.

For an internal CA, Node clients need NODE_EXTRA_CA_CERTS=/path/to/internal-ca.pem, set before the process starts. Both Python SDKs now run on httpx2, which OpenAI’s migration notes say checks certificates against the operating system’s store. To pin your CA in code, hand the SDK’s own client class an SSL context:

import ssl
from openai import OpenAI, DefaultHttpx2Client

ctx = ssl.create_default_context(cafile="/path/to/internal-ca.pem")
client = OpenAI(
    base_url="https://llm-gateway.example.internal/v1",
    http_client=DefaultHttpx2Client(verify=ctx),
)

With Anthropic’s SDK the class is DefaultHttpxClient, imported from anthropic. Its documentation says a client built from the separate httpx package raises a TypeError.

Checking that personal data leaves masked

Before the first real client record goes through, send a made-up one and ask the model to repeat it. Your application should get the real values back, and the audit log or dashboard should show the request with a count of what was masked, by type. Counts only, never the data.

curl -sS http://127.0.0.1:8080/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-5.5", "messages": [{"role": "user", "content": "Copy exactly: Ruairi Calloway, +1 202 555 0183, ruairi.calloway@example.com, SSN 987-65-4320"}]}'

A test using only neat English names proves very little. Build the set from what your business really sends, starting from our checklist of the details worth removing. Per data type you choose mask, block or flag, and if detection fails the request is not sent by default. I’d leave that default alone.

If you want to see the outgoing request exactly as the provider receives it, don’t improvise by repointing the upstream. Ask us, or check the install notes, for a supported way to do that in a test set-up.

Container logs or the audit log: which record for whom

Two records come out of a running gateway, and they answer different people. docker compose logs is for operations: start-up, warnings, errors. The Gateway’s audit log is for your DPO: one tamper-evident line per request with provider, model and how many items of each type were found, exportable as CSV or JSONL.

Docker’s json-file driver keeps logs without a size limit unless you set one, hence the cap of five 20 MB files. The bigger risk is downstream: if proxy logs reach a central platform with request bodies switched on, prompts sit there readable. It is a quiet way client data escapes through AI tooling.

Your client list lives on the same server. Export clients and companies from your CRM as CSV or Excel, load the file into the Gateway, and reload it when it changes, without restarting the container. A list that is a month out of date is the gap I’d watch most closely.

Updating a pinned image and rolling back

The Gateway never updates itself. It tells you a new version exists, and your team decides when to apply it:

# change the tag in GATEWAY_IMAGE inside .env, then:
docker compose pull gateway
docker compose up -d gateway
docker compose logs --tail 20 gateway

Rerun the masking test after every update; it takes a couple of minutes. If anything looks off, put the previous tag back in .env and run docker compose up -d gateway again. Unlike a library you assemble yourself, new versions arrive ready-made rather than as a maintenance job for your team.

With one instance, recreating the container means a few seconds without service. For an internal chatbot that is usually fine; for an agent pipeline running overnight, I’d rather have two copies behind a load balancer and move them one at a time. Pick your layout below for the order of the steps.

Troubleshooting an AI gateway in Docker

Most failures land in one of five places: the image pull, the start, the request, TLS or streaming. Filter the table to the stage where yours stopped. I trust the stage more than the error text: the same 401 means different things depending on who sent it.

If the stage is unclear, run the curl test from the host. Working there but not from a service puts the fault in nginx, the firewall or the client. Once the wiring is proven, an AI policy staff can actually sign can name the gateway as the only route to the API.

What leaves your network once the AI gateway is live

Once the masking test passes and TLS is in place, the picture is simple. Detection runs on your server, the link from each placeholder back to the real value stays there, and none of your content passes through Nonimo. If you go over your plan’s monthly volume, the Gateway tells you and keeps running.

With a hosted gateway, that inspection runs on the vendor’s network instead, as our look at its DLP profiles for AI traffic explains. For me, that is the whole case for running a container yourself: the inspection happens on a machine your team patches and your DPO can point at.

When the test server has done its job, choose a Gateway plan with room for the instances your layout needs, and the licence email gets the first production host going.

Sources

• Docker: packet filtering and firewalls: traffic to published container ports bypasses ufw rules.
• Docker: JSON File logging driver: max-size defaults to unlimited and max-file to 1.
• nginx: ngx_http_proxy_module: proxy_buffering on and proxy_read_timeout 60s by default.
• nginx: client_max_body_size: 1m by default, 413 when exceeded.
• OpenAI Python SDK README: base_url, OPENAI_BASE_URL and a custom DefaultHttpx2Client with verify.
• OpenAI Python SDK: migrating to HTTPX2: HTTPX2 uses the operating system’s trust store.
• Anthropic: Python SDK: ANTHROPIC_BASE_URL, DefaultHttpxClient, and a TypeError for clients from the old httpx package.
• Node.js: NODE_EXTRA_CA_CERTS: extra trusted CAs, read only at process start.
• Python: ssl.create_default_context: loading a CA file for certificate verification.

Accessed 10 November 2026.