Local and private
What is local AI?
Local AI runs a model on hardware you control. What it changes about privacy and cost, what it demands in hardware, and where it genuinely competes.
All explainers · Last reviewed:
What it changes
Data is the real reason. A prompt to a hosted model is processed by that provider under their terms. A prompt to a local model is not sent anywhere, which is the only version of that guarantee that does not depend on someone else’s policy.
It also removes per-token billing, rate limits, and the possibility of a model being retired underneath you. A model you have downloaded keeps working.
The cost shape, not the cost
Local AI is not automatically cheaper. It converts a variable cost into a fixed one: you buy or rent hardware, and then inference is close to free at the margin.
That is a good trade at high, steady volume and a bad one for occasional use. A GPU idling most of the day is more expensive than the API calls it replaced.
The honest limits
Models that run comfortably on a single machine are generally not the largest models available. On the hardest reasoning tasks the gap between a local model and a frontier hosted model is real.
For a great many everyday tasks — summarising, drafting, extracting, classifying, routine code — the gap is much smaller than people assume, and the privacy and cost properties often matter more than the last increment of capability.
Most useful as a hybrid
The common pattern is not local-only or cloud-only. It is local for anything sensitive or high-volume, hosted for the hardest questions, and a policy deciding which is which — which is exactly what a router is for.
Questions people ask
- What hardware do I need?
- It depends entirely on model size and quantisation, and anyone who gives you a single number is guessing. The dominant constraint is available memory: the model’s weights have to fit, and what fits determines what you can run.
- Is local AI private by definition?
- The model call is. The rest of the application may not be — search, telemetry and other integrations can still reach outside. Privacy is a property of the whole system, not of one component.
- Can local models use my documents?
- Yes. Retrieval works the same way, and when both the retrieval and the model are local the documents never leave your hardware at any point.
Try it rather than read about it
ClawAI runs local models through Ollama and llama.cpp, and its local-only routing mode keeps the whole fallback chain on local providers rather than reaching for a cloud model.