Ollama vs llama.cpp: which should you use?

What each one is

llama.cpp is a C++ inference engine. It runs quantised models efficiently on CPUs and GPUs, and it exposes fine-grained control over how a model is loaded and executed. It is the lower layer, and much of the local-AI ecosystem is built on it.

Ollama wraps that kind of engine in convenience: pull a model by name, run a server, get an HTTP API, have model files and memory managed for you. It optimises for getting a model running in a minute.

How to choose

Choose Ollama when you want models running quickly with sane defaults, when you will switch between several models, or when you want a stable local API without tuning anything.

Choose llama.cpp directly when you need control — specific quantisation, specific layer offloading, unusual hardware, or embedding inference into your own binary. The cost is that you manage the details yourself.

Using both is normal

A common arrangement is Ollama for everyday interactive use and llama.cpp for a workload that has been tuned deliberately. They are not mutually exclusive, and a platform that supports both lets the decision be made per deployment rather than once.

Questions people ask

Is Ollama just a wrapper?

That undersells it. Model management, memory handling and a consistent API are the parts that make local models practical day to day, and they are genuine work regardless of what engine is underneath.

Which is faster?

On the same model, quantisation and hardware, they are close, because the heavy lifting is the same kind of work. Differences in practice usually come from configuration rather than from the tool.

What is quantisation?

Storing model weights at lower precision so they need less memory. It is what makes large models fit on ordinary hardware, and it trades a small amount of quality for a large amount of practicality.