Before your cloud LLM API bill starts to scare you — Running a dedicated on-premises AI server for your team with Ollama + Open WebUI

When a development team adopts an LLM, it looks fine at first. You just get an API key and start making calls. But as the team grows and usage increases, two major problems tend to surface at the same time.

The first is cost. API bills running hundreds of thousands of dollars a month become a fixture on the budget meeting agenda. The second is data. The fact that queries containing customer information or internal documents are being sent to external clouds starts showing up on the radar of legal and security teams. With the EU AI Act coming into full effect in August 2026 and GDPR compliance concerns added on top, the conversation naturally turns to "wouldn't it be better to run this ourselves instead of using an API?"

There is one more issue: team members are already using ChatGPT or cloud APIs for work on their own — what's known as Shadow AI. Rather than leaving uncontrolled Shadow AI data flows unchecked, it's far more advantageous from both a security and compliance standpoint to absorb them into official infrastructure through an in-house LLM server.

This post covers the experience of building a team-shared AI infrastructure using Ollama and Open WebUI. It goes beyond "just run this command" to address the real challenges you face in production: model management strategy, role-based access control (RBAC) design, and setting up a GPU usage monitoring pipeline. It also flags the pitfalls to avoid — like accidentally exposing the Ollama API port to the internet, or silently breaking the service when VRAM fills up.

The target audience is backend and DevOps engineers. It's best suited for those already comfortable with Docker and basic Linux commands who have a GPU server in hand.

Core Concepts

The Big Picture First

It's much easier to understand the details once you see the overall structure.

The in-house LLM server is organized into four layers.

• Ollama — the model serving backend. Exposes an OpenAI-compatible REST API on localhost:11434.
• Open WebUI — the authentication, user management, and chat UI layer, sitting on top of Ollama.
• Nginx — the reverse proxy, handling TLS termination and external traffic routing.
• Prometheus + Grafana — monitors GPU usage and inference performance.

The Ollama API is never exposed directly to the outside — access must always go through the Open WebUI authentication layer. This principle is the foundation of the security design.

Ollama: A Runtime That Spins Up Models with One Command

Ollama's greatest appeal is its low barrier to entry. It handles model downloading, quantization, and GPU/CPU resource allocation automatically, so you can run Llama 3.3 70B locally with just this one line:

ollama run llama3.3:70b

It exposes an OpenAI-compatible API, so existing code only needs the endpoint URL changed. However, connecting directly to the Ollama API is for local development environments only.

# For local development only — use the Open WebUI endpoint on team servers
from openai import OpenAI
 
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",
)
response = client.chat.completions.create(
    model="llama3.3:70b",
    messages=[{"role": "user", "content": "Please review this code"}],
)

In a team server environment, use Open WebUI's OpenAI-compatible endpoint together with an API key issued by Open WebUI. API keys can be generated in the Open WebUI admin panel → Settings → Account.

# Team server environment — authenticated with an Open WebUI API key
from openai import OpenAI
 
client = OpenAI(
    base_url="https://llm.yourcompany.internal/api",
    api_key="sk-...",  # Issued from the Open WebUI admin panel
)
response = client.chat.completions.create(
    model="llama3.3:70b",
    messages=[{"role": "user", "content": "Please review this code"}],
)

Note: Ollama handles roughly 4 parallel requests by default. As team size grows, you may want to consider switching to vLLM or TGI (Text Generation Inference).

Open WebUI: Authentication and UI in One

Open WebUI provides a ChatGPT-like web interface while bundling the features a team actually needs.

Practical Implementation

Step 1: Bringing Up the Stack with Docker Compose

This setup assumes NVIDIA Container Toolkit is installed on the GPU server. Before starting, create a .env file first.

echo "WEBUI_SECRET_KEY=$(openssl rand -hex 32)" > .env

# docker-compose.yml
services:
  ollama:
    image: ollama/ollama:0.5.13  # Pin to a specific release tag — check hub.docker.com
    container_name: ollama
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"   # Bind to localhost only — do not expose externally
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
 
  open-webui:
    image: ghcr.io/open-webui/open-webui:v0.5.16  # Pin to a specific release tag — check GitHub Releases
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "127.0.0.1:3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY}
      - WEBUI_AUTH=true
      - DEFAULT_USER_ROLE=pending   # Set new user default to Pending
    volumes:
      - open_webui_data:/app/backend/data
    depends_on:
      - ollama
 
volumes:
  ollama_data:
  open_webui_data:

The binding to 127.0.0.1:11434 is the critical part. Binding to 0.0.0.0:11434 allows anyone to call models and occupy VRAM without authentication. The official Ollama image (ollama/ollama) automatically uses the GPU in environments with NVIDIA Container Toolkit installed — a separate :cuda tag is not needed.

Pin image versions to specific release tags rather than :latest or :main to enable reproducible deployments and rollbacks. The version numbers above are examples — check each repository's latest release before deploying.

Step 2: Nginx + TLS Configuration

# /etc/nginx/sites-available/llm-server
server {
    listen 443 ssl http2;
    server_name llm.yourcompany.internal;
 
    ssl_certificate     /etc/ssl/certs/llm-server.crt;
    ssl_certificate_key /etc/ssl/private/llm-server.key;
 
    location / {
        proxy_pass         http://127.0.0.1:3000;
        proxy_http_version 1.1;
        proxy_set_header   Upgrade $http_upgrade;
        proxy_set_header   Connection "upgrade";
        proxy_set_header   Host $host;
 
        proxy_buffering    off;   # Without this line, streaming responses won't work
        proxy_cache        off;
        proxy_read_timeout 300s;
    }
}
 
server {
    listen 80;
    server_name llm.yourcompany.internal;
    return 301 https://$host$request_uri;
}

Without proxy_buffering off, streaming token responses arrive all at once in a single chunk. If you're wondering why the typing effect isn't appearing, this is the first thing to check.

SSL Certificate Issuance: For an internal-only server, you can set up a local CA with mkcert or issue a self-signed certificate with openssl.

# Option 1: mkcert — installs a local CA on team PCs, no browser warnings
mkcert -install
mkcert llm.yourcompany.internal
 
# Option 2: openssl self-signed — requires adding a certificate exception in the browser
openssl req -x509 -newkey rsa:4096 -keyout llm-server.key \
  -out llm-server.crt -days 365 -nodes \
  -subj "/CN=llm.yourcompany.internal"

Step 3: Model Management Strategy

Before designing RBAC and deciding "which groups can access which models," you first need to decide which models to run.

# Download models
ollama pull llama3.3:70b
ollama pull qwen2.5-coder:32b
ollama pull mistral:7b
 
# Check models currently loaded in memory
ollama ps
 
# List available models
ollama list

For a team-shared server, it's useful to create custom models with a fixed system prompt via a Modelfile. Below is an example of a dedicated code review model. Modelfile syntax is similar to Dockerfile but is an Ollama-specific format.

# Modelfile.codereview  (Ollama Modelfile format)
FROM qwen2.5-coder:32b
 
SYSTEM """
You are a senior backend engineer. When asked to review code, you provide specific feedback on security vulnerabilities, performance issues, and readability problems.
You respond in English and explicitly quote the problematic lines.
"""
 
PARAMETER temperature 0.3
PARAMETER num_ctx 8192

ollama create codereview -f Modelfile.codereview

Register this model in Open WebUI as a team-wide shared model so all team members can use the same code review assistant with a consistent context.

VRAM Planning: Loading a 70B model with Q4 quantization requires approximately 40 GB of VRAM. Limit the number of models loaded at all times to fit your server's VRAM capacity, and load infrequently used models on demand.

Step 4: RBAC Design — Understanding the Additive Permission Model

Open WebUI's permission model is additive. Group membership only adds permissions — it never removes them. Understanding this principle prevents design mistakes.

The design principle is simple: set default permissions as low as possible and elevate them via groups. In this diagram, "set default role to Pending" is controlled by the Docker Compose environment variable DEFAULT_USER_ROLE=pending from Step 1. The same value can also be changed in the Open WebUI admin panel (Admin Panel → Settings → General → Default User Role).

Base role structure:

A common real-world configuration has the Admin manually promote new signups from Pending → User, with different model access configured per group (e.g., engineering team vs. data team). Advanced integrations like LDAP group-based auto-approval vary in support across Open WebUI versions — check the official documentation and the repository's latest release notes. For enterprise IdP integration, OIDC/OAuth2 or SCIM 2.0 is recommended.

Step 5: GPU Monitoring Pipeline

Deferring GPU monitoring until later means you'll find out too late when VRAM fills up or temperature exceeds thresholds. Latency can silently increase by orders of magnitude with no other symptoms, so it's worth setting this up from the start.

Ollama does not have a /metrics endpoint that Prometheus can scrape directly. GPU metrics are collected via NVIDIA DCGM Exporter, and Ollama status metrics via a separate sidecar exporter.

The sidecar exporter can be implemented as a simple Python script.

# ollama-exporter/main.py
import time
import requests
from prometheus_client import start_http_server, Gauge
 
OLLAMA_URL = "http://ollama:11434"
 
loaded_models_count = Gauge(
    "ollama_loaded_models_total",
    "Number of models currently loaded in memory"
)
 
def collect():
    while True:
        try:
            resp = requests.get(f"{OLLAMA_URL}/api/ps", timeout=5)
            if resp.ok:
                models = resp.json().get("models", [])
                loaded_models_count.set(len(models))
        except Exception:
            loaded_models_count.set(0)
        time.sleep(15)
 
if __name__ == "__main__":
    start_http_server(8000)
    collect()

# ollama-exporter/Dockerfile
FROM python:3.12-slim
RUN pip install prometheus-client requests
COPY main.py .
CMD ["python", "main.py"]

For per-request metrics such as P99 latency and token throughput, the recommended approach is to place a FastAPI middleware proxy in front of the Ollama API and instrument it there. That's outside the scope of this post and will be covered separately.

Add the entire monitoring stack to Docker Compose.

# Add to docker-compose.yml
  ollama-exporter:
    build: ./ollama-exporter
    container_name: ollama-exporter
    restart: unless-stopped
    ports:
      - "127.0.0.1:8000:8000"
    depends_on:
      - ollama
 
  dcgm-exporter:
    image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.5-3.4.0-ubuntu22.04  # Check NGC for the latest tag
    container_name: dcgm-exporter
    restart: unless-stopped
    ports:
      - "127.0.0.1:9400:9400"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    cap_add:
      - SYS_ADMIN
 
  prometheus:
    image: prom/prometheus:v2.53.0  # Pin to the latest tag
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "127.0.0.1:9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
 
  grafana:
    image: grafana/grafana:11.2.0  # Pin to the latest tag
    container_name: grafana
    restart: unless-stopped
    ports:
      - "127.0.0.1:3001:3000"
    volumes:
      - grafana_data:/var/lib/grafana
 
volumes:
  prometheus_data:
  grafana_data:

# prometheus.yml
global:
  scrape_interval: 15s
 
scrape_configs:
  - job_name: dcgm
    static_configs:
      - targets: ["dcgm-exporter:9400"]
 
  - job_name: ollama_sidecar
    static_configs:
      - targets: ["ollama-exporter:8000"]

Here's a summary of the metrics flow.

Key panels to watch in Grafana:

Trade-off Analysis

Cloud API vs. On-Premises Comparison

A word on break-even point: Specific figures like "$5,000/month in API costs" or "100 million tokens/month" are frequently cited, but these are examples based on specific assumptions about GPU rental rates, API model pricing, and engineering labor. Those ranges come from calculating with an A100 80GB GPU at roughly $2,000/month and GPT-4o input pricing — but they vary greatly depending on server specs, contract type, and usage patterns. A real TCO calculation must include engineering labor, security patch response, and monitoring operation time. When these are factored in, the total cost often ends up significantly higher than a simple infrastructure-only calculation suggests.

When On-Premises Makes Sense

• Processing sensitive data (customer information, medical or legal documents) with an LLM
• Current API costs are high enough that the ROI of switching is clearly calculable
• Environments requiring GDPR, HIPAA, or personal data protection law compliance
• AI services needed on air-gapped (internet-disconnected) internal networks
• Shadow AI needs to be brought under governance

Common Mistakes in Practice

Mistake 1: Exposing the Ollama Port Directly to the Internet

Binding to 0.0.0.0:11434 lets anyone call models and occupy VRAM without authentication. Always use 127.0.0.1:11434:11434 in Docker Compose for a localhost-only binding, and only allow external access through the Nginx + TLS + Open WebUI authentication layer.

Mistake 2: Failing to Plan for VRAM

Loading a 70B model with Q4 quantization requires approximately 40 GB of VRAM. When multiple team members send requests simultaneously, KV cache accumulates and drains VRAM — and if another model is loaded at that point, the existing model gets swapped out and latency spikes dramatically. Set up a VRAM >85% alert with DCGM Exporter and limit the number of models kept loaded at all times to fit within VRAM capacity.

Mistake 3: Starting with Overly Broad Default Permissions in RBAC

Because Open WebUI permissions are additive, setting broad User permissions from the start makes it difficult to restrict specific groups later. Start with DEFAULT_USER_ROLE=pending and design so that permissions are added at the group level as needed.

Mistake 4: Calculating TCO Using Only Infrastructure Costs

Looking only at GPU rental costs can make it feel like it's "much cheaper than cloud APIs," but when engineering labor, model updates, security patches, and monitoring response are added in, the true total cost far exceeds the bare infrastructure figure. Always include team labor costs in the calculation before deciding to make the switch.

Mistake 5: Omitting proxy_buffering off

Without this single line in the Nginx configuration, streaming responses will not work. It will appear as though responses only appear after generation is complete, which degrades the perceived performance.

Closing Thoughts

With a GPU server with 24 GB or more of VRAM and internal DNS in place, the Ollama + Open WebUI combination lets you build a team-scale AI infrastructure. Compared to cloud LLM APIs, you gain three core advantages: data sovereignty, cost predictability, and vendor independence. The benefits are most pronounced for teams handling sensitive data or whose API costs have exceeded the ROI threshold.

Here's a three-step approach you can start on today:

Step 1 — Local Validation: Spin up Ollama + Open WebUI with docker compose up -d on a development machine with a GPU, and directly evaluate open model performance on tasks your team uses frequently (code review, document summarization, internal FAQ responses, etc.). Llama 3.3 70B and Qwen 2.5 72B produce practical results for routine in-house tasks like coding assistance and document summarization.

Step 2 — Team Pilot: Set up Nginx + TLS on a shared GPU server and run a 2–4 week pilot with a small group (5–10 people). RBAC design and model selection criteria will naturally take shape through this process.

Step 3 — Add Monitoring: Connect DCGM Exporter + sidecar exporter + Prometheus + Grafana and set up a VRAM >85% alert. Without this, you'll find out about problems after the fact.

If your team members are already using Shadow AI for work, the longer you delay absorbing it into official infrastructure, the larger the data governance gap grows. The initial setup takes some effort, but once it's running, the whole team ends up getting real value out of it.

References

• Open WebUI Official Docs — Authentication & Access Control
• Open WebUI Official Docs — RBAC Overview
• Open WebUI Official Docs — Role Definitions
• Open WebUI Official Docs — Permission Management
• Open WebUI Official Docs — SSO
• GitHub — open-webui/open-webui Official Repository
• Open WebUI Sovereign AI Platform
• Monitor Ollama with Prometheus & Grafana: Production Dashboards
• Monitoring Self-Hosted LLM with Prometheus and Grafana — DEV Community
• GPU Monitoring: Monitoring GPU Memory and Inference Throughput with Prometheus & Grafana
• Ollama Production Monitoring: Practical Guide to Logging and Prometheus Alerts
• Local LLM Ops: Building an Observable GPU-Accelerated AI Cloud at Home with Docker & Grafana — DEV Community
• The Complete Ollama Enterprise Deployment Guide
• Self-Hosted LLM vs API: Enterprise Cost & Security
• Self-Hosted LLM Guide: Costs, Architecture & Breakeven Point
• Ollama Production Deployment: Docker-Compose Configuration Guide
• Ollama in Production: Docker, SSL, Auth & Monitoring
• SUSE AI 1.0 — Open WebUI User Account Management Official Docs