How Your AI Homelab Scales Into National Infrastructure

Self-hosted AI is easy to dismiss as a homelab hobby.

That misses the important part: the choices you make on one machine are the same ones larger organizations have to make later.

The software can stay surprisingly familiar. What changes is everything around it: access, uptime, support, governance, and accountability.

At home, one person decides where the model runs, who can use it, what data stays local, and what rules apply. In a larger organization, those decisions come with datacenters, identity systems, procurement, policy, and support teams.

Self-hosting is one part of what people call sovereign AI, the broader question of who controls an AI system end to end. It's the part that fits on a sticky note: you run the AI yourself, so you decide where it runs, which models it can reach, and where your data goes.

Open WebUI and a local model, running on one machine

The same setup, at any scale#

At the small end, it's one person with a machine and a local model. They install a runtime like Ollama to run the model itself, put Open WebUI in front of it as their everyday interface, and chat with a model that lives entirely on that machine.

For a lot of people, that's the whole thing, and it never needs to be more.

In the middle are the organizations that can't send their data anywhere else. Think of a company bound by contract to protect customer records, a hospital with patient files, a team whose network never touches the internet, or a university standing up a shared assistant for staff and students.

Whatever the reason, the goal is the same: keep the AI, its users, and its data on systems they control.

At the far end, the same setup shows up in public-sector deployments.

According to press reports, one public-sector rollout gives roughly 180,000 federal employees a self-hosted AI workspace built on open-weight Mistral models. The models run in a federal datacenter, and Open WebUI serves as a user interface.

The announced plan grows that to roughly 250,000 employees across the wider public sector by the end of 2026.

That is the point: self-hosted AI is not just what you do when you want to tinker with a GPU at home. It is also what you do when the organization is too large, too regulated, or too accountable to let someone else own the path between users, models, and data.

Lessons from the far end#

A public-sector deployment like that makes the checklist easy to see: where the models run, who operates them, what interface people use, and whose rules apply.

The same questions show up at home.

The answers are just smaller: your machine, your account, your interface, your rules.

Inside an organization, those answers need names attached. Someone owns access. Someone owns uptime. Someone owns support. Someone owns the policy.

The full playbook behind those decisions, from definitions to stack layers, lives in our sovereign AI guide.

A practical scale ladder#

Find your row. Moving down the table mostly means adding hardware and switching on more of what's already there.

What hardware you actually need#

Rules of thumb, not laws:

• To try it, use what you own. A small model runs CPU-only in 16 GB of system memory, with 32 GB giving it more room to breathe. Responses will be on the slow side, which is fine while you're finding out whether self-hosting suits you. If it becomes a daily tool, a GPU is the first upgrade worth making.
• For GPU inference, 12 GB of VRAM is the practical entry point. It runs 7B to 14B models well at common quantizations, the compressed formats most local models ship in. 24 GB opens up much more capable local models. Beyond that, you're shopping for a multi-GPU rig or a dedicated server, and you're usually building for a team by then, not for yourself.
• Apple Silicon can be a good homelab path. Unified memory lets a Mac run models that would need a serious GPU elsewhere.
• Buy hardware last. Decide the model size, response speed, user count, and privacy boundary you actually need, then size the machine to that. Doing it the other way around is a common and expensive self-hosting mistake.

Starting from zero#

You don't need a datacenter to start. Two programs on one machine will do, and getting there takes four steps.

First, install Ollama, which keeps running in the background from then on, and pull a model, for example:

ollama pull qwen3:8b

Second, run Open WebUI next to it as a single container:

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

If you'd rather not use Docker, Open WebUI installs with pip (Python 3.11 or newer) and lands at http://localhost:8080 instead:

pip install open-webui
open-webui serve

Kubernetes is an option too if that suits your environment better.

Third, open http://localhost:3000, or 8080 if you went the pip route. The first screen asks you to create an admin account, which is saved on your own machine and nowhere else. You're the administrator from day one, even when you're the only user.

The docker command above includes the flag that lets the container reach programs on your machine, so Open WebUI connects to your Ollama automatically and its models show up in the selector.

Pick one and you're talking to an AI that runs entirely on your own hardware.

A fresh install's first screen. Get started leads straight to creating your local admin account.

Fourth, if no model appears, it's usually one of two things.

Either Ollama isn't running (ollama list in a terminal should answer instantly), or the connection needs pointing at it. In admin settings, open Connections and check that the Ollama address reads host.docker.internal:11434.

If you've put an API key on the connection, check that too, since a swapped or expired key shows the same empty model list.

Future models don't need the terminal. Type a name into the model selector and Open WebUI offers to pull it from Ollama for you, progress bar included.

Pulling a small model straight from the model selector, from search to done

At this point you have a working AI that's entirely yours, on hardware you already owned. Later, when you want them, you add hosted connections, invite other people, and climb the ladder at your own pace.

One interface, any model#

One interface, any model, local or hosted

Open WebUI talks to local runtimes like Ollama and vLLM and to hosted APIs like OpenAI and Anthropic through the same screen. Your model list can hold a small model on your own GPU today and a top-tier hosted API tomorrow, and nobody's workflow changes in between. It also means a hosted model is one connection away when it's the better tool for the job.

When a hosted model is the right call#

Self-hosting isn't a purity test. Hosted frontier models are the most capable models you can use today. If your work needs the strongest reasoning available, if your usage is too occasional to justify hardware that sits idle, or if the data involved isn't sensitive, then ChatGPT, Claude, or another hosted service is a sound choice on its own merits.

Plenty of setups land on a mix: a local model for anything confidential and a hosted one for the hard problems. Running your own interface means the call stays yours, per task.

The first controls to turn on#

Once other people join your setup, the controls come in a natural order:

1. Accounts. The day a second person joins, turn on admin approval for new sign-ups, so access is something you grant rather than something that happens.
2. Groups and per-model access. Each team sees the models it's meant to use, and your expensive or sensitive endpoints stop being available to everyone by default.
3. Single sign-on. Plug in the company login your team already uses. OAuth and OIDC handle the sign-in, and accounts stop being something you manage by hand.
4. Directory, logs, and retention. SCIM provisioning, audit logs, a retention policy, and a tested backup stop being optional in a hospital or a ministry.

Steps one and two in practice. A sign-up sits as Pending until an admin approves it, and groups scope what their members can reach.

Stop at step one if that's all your setup needs, and go further only when the next control is actually needed. The reasoning behind the order is in the sovereign AI guide.

Common mistakes#

• Sizing storage for the models and forgetting the data. Model files are the predictable part, while chats, uploads, and the search indexes built from them keep growing.
• No backup until the first migration scare. It's one volume. Schedule a copy, and restore it once to prove it works.
• No retention decision. Decide early how long chats and logs stick around. It's a cheap decision now, and it gets harder once someone's workflow depends on the archive.
• Wiring in a hosted API and exposing it to everyone. Scope it with per-model access on day one rather than as an afterthought.
• Assuming self-hosting equals compliance. It doesn't. Where data lives and who touched it become questions you can answer inside systems you operate, but whether that satisfies GDPR or the EU AI Act depends on how you run the whole thing. The software gives you control; it does not make the legal judgment for you.

The hosted-API mistake, fixed. A hosted model set to Private and scoped to one group instead of open to everyone.

The honest tradeoff#

Self-hosting swaps a subscription for responsibilities. Read the full bill before you commit.

The hardware needs buying, sizing, and keeping online. The software needs updates, security patches, and backups you've actually tested. And the people questions need owners: who reads the logs, how long anything is kept, who supports the users, and who makes the compliance decisions.

The bigger the setup gets, the more of that bill becomes someone's actual job, starting with a plan for something as small as a reboot.

In exchange, the important parts stay under your control: the models, the data, the interface, and the rules.

Get involved#

• Run it yourself. The four steps above are one evening. Start at the docs.
• Read the full playbook. Definitions, stack layers, and rollout order live in our sovereign AI guide.
• Tell us how you deploy. Homelab, company, hospital, campus, or public sector, we want to hear how Open WebUI is being run, and at what scale. Share it with the community.

The Open WebUI Team