Where AI Actually Runs
Everyone argues about which model to use. The more interesting question is where it should run.
A mechanical watch keeps perfectly good time on your wrist, answerable to nothing and connected to nobody. Most of our software has forgotten that self-sufficiency was ever an option.
Everyone asks which model to use. Almost nobody asks where it should run, which is the more interesting question and, for anyone who makes hardware, the one that pays the bills. I spent a stretch on this for a hardware client, mapping where AI workloads belong on the device in your hand against in a data centre somewhere, across a long list of industries. The honest first finding was that "just use the cloud" is about half an answer. It is the right default for the largest models and the heaviest jobs, and quietly wrong for a growing share of real cases. The line between device and cloud is not a principle. It is a snapshot, and it keeps sliding toward the device.
“The line between device and cloud is not a principle. It is a snapshot, and it keeps sliding toward the device.” THE CENTREPIECE CLAIM
Here is the frame I kept returning to.
The four things that actually decide it
Strip away the noise and the question comes down to four axes. Most use cases are an argument among them.
Latency. How fast must the answer come back, and can it survive a round trip to a server? Drafting an email, who cares, the cloud is fine. A real-time assistant on a factory line or in a vehicle cannot wait on a network hop and a busy server, so it leans local.
Privacy and data residency. Where is the data allowed to go? Some of it cannot leave the building, the device, or the country, for reasons of regulation, contract, or plain sense. If sending the input to someone else's servers is a problem, the model has to travel to the data rather than the reverse.
Cost at scale. Cloud inference is rented by the use, which is wonderful at low volume and brutal at high, because you pay per call forever. On-device inference has the opposite shape: the hardware is a sunk cost and the marginal call is nearly free. Past a certain volume, local stops being a constraint and becomes a saving.
Connectivity. Must it work when the network does not? Anything used in the field, underground, on a plane, or anywhere with a weak signal cannot depend on the cloud being reachable. Offline is not a niche. It is most of the physical world on a bad day.
No single axis settles it. You weigh all four, and the weighting flips entirely depending on what the thing is for.
Why the frontier is pushing work toward the device
For a while the device side of this was theoretical, because anything useful was too large to run outside a data centre. That is changing fast, for three reasons worth understanding.
Models are getting smaller without getting proportionally dumber. A model a fraction of the size of the frontier flagship is now good enough for a wide class of practical tasks, and "good enough, locally, instantly" beats "excellent, in the cloud, eventually" more often than the benchmark-chasers admit.
Quantization makes those models cheaper to run. Roughly, you trade a little numerical precision in the model's weights for a large drop in memory and compute, so a model that would never have fit on a laptop or a phone now does, at a quality cost small enough to ignore for most uses.
And the hardware grew dedicated help. Modern devices increasingly ship with accelerators built for exactly this kind of arithmetic, which is the whole reason a hardware client cares about the question at all. When the silicon in your hand can do the work, the sums for what belongs on-device shift under everyone's feet.
A distinction that changes the hardware question
One framing I borrowed and kept: the difference between retrieval-style compute and generative compute.
Looking something up, matching, ranking, pulling the relevant fact, is comparatively cheap and a strong candidate for running locally. Generating something new, a long document, an image, a fresh chain of reasoning, is far heavier and more often wants the big models in the cloud. A lot of "should this run on-device" questions resolve cleanly the moment you ask which kind of work it really is. Plenty of products assume they need generative horsepower when most of what users do is retrieval, and that single mistake sends workloads, and cost, to the cloud that never needed to go.
How it shakes out by industry
Run real use cases through those axes and patterns appear rather than a single answer.
The regulated and sensitive industries, healthcare, legal, finance, lean on-device wherever they can, because the privacy axis dominates everything else. The data is the whole problem, and the cleanest way to protect it is to never let it leave.
The latency-critical and offline industries, manufacturing, field engineering, anything operational and physical, also lean local, but for latency and connectivity rather than privacy. The work happens in a place and at a speed the cloud cannot reliably serve.
The heavy generative work, large creative production, big synthesis, the largest-model tasks, stays in the cloud, because cost and capability still favour renting a data centre over carrying one.
And a vast middle ground goes either way, decided by volume and by whatever hardware the user already holds. That middle is exactly where the boundary is moving, and where building a sixty-cell taxonomy mostly taught me that the honest answer to a lot of the cells is "it depends, and here is on what."
The useful version for someone deciding
If you are choosing for a real product, the shortcut is this. Start from the data and the latency. If the data cannot leave or the answer cannot wait, you are on-device, and the rest is engineering. If neither constrains you, default to the cloud for capability and revisit once your volume grows enough that the per-call cost starts to sting. Then check whether you are doing generative work or merely expensive retrieval, because that one question reroutes a surprising number of workloads.
The thing to hold onto is that none of this is settled. Every year the small models get more capable, quantization gets better, and the accelerators in ordinary devices get stronger, so the boundary creeps steadily toward the edge. The task that obviously belongs in the cloud today is a strong candidate to run in your pocket a year or two from now, quietly, answerable to nothing, the way a good watch always has. ▮