Running AI on your own servers: what actually works

The real question is not technical

When an owner asks “can we run AI in-house?”, they are not asking an infrastructure question. They are asking: will my data go to someone else, and will I know about it?

The technical answer is simple. The honest answer requires separating three things that are systematically conflated.

What “running AI” covers

Reading and filing. Sorting mail, recognising a document type, extracting an amount or a date, summarising a page. This is the vast majority of real business use, and it is the least demanding part. Small models, installed on an ordinary machine, do this work properly.

Reasoning on a complex case. Analysing a contractual situation, comparing scenarios, writing text that must persuade. Here large models keep a clear lead, and they run poorly on standard business hardware.

Writing code. A special case, very demanding, almost always handled by remote models today.

Conflating these three is the most common mistake. It leads either to oversizing a machine for mail sorting, or to sending documents to the cloud that had no reason to leave.

The architecture that works

Sort first, escalate second.

Everything incoming passes through a local model. It handles what it can handle — and it can handle a lot. What it cannot handle, or is unsure about, it flags. Only then, and only for those cases, may the request go to an outside service.

That escalation point is where sovereignty is decided. It is where you write the rule: which document types may leave, which never may, and what is stripped from the text before sending. A written rule, reviewed, and verifiable in the logs.

In practice, a large share of the volume never leaves.

What hardware it takes

Less than imagined for the first category. A dedicated machine with a mid-range graphics card is enough to process a company mail flow and query a document base.

The calculation to make is not “what does the machine cost”, but “what does the machine cost, amortised over three years, against three years of subscription at the volume I consume”. That calculation leans local more often than people expect, especially when volume is steady. It leans cloud when usage is sporadic.

The traps

Believing a local model is “private” by magic. It is private because it runs on your premises, not because it is called local. If your application still sends a copy elsewhere for tracking or metrics, you have gained nothing. Check what leaves, not what is claimed.

Underestimating operations. A machine on your premises is a machine to back up, update and monitor. That is not hard, but it is not free either, and you should decide who handles it before ordering it.

Starting from strategy. The right way to start is a single measurable use: one document type, one mailbox, one recurring question. Run it for real for a month, measure the time saved, and widen only if the figure justifies it.