Small models, big effects: AI that runs on your own machine

Generative AI first brings to mind data centres and clusters of GPUs. That is true of the largest models. But part of the useful work is done very well by much smaller models, with a few billion parameters, that run on a laptop, offline and without a per-token bill. Understanding them lets you choose the right tool instead of defaulting to the biggest one.

Why size is counted in memory

A model is a very large array of numbers, the parameters or weights. To run it, it has to be loaded into memory. A simple calculation gives the order of magnitude: a seven-billion-parameter model stored with two bytes per parameter, which is common in half precision (sixteen bits), takes up about fourteen gigabytes. That is too much for most laptops.

On top of that comes the context memory (the KV cache), which grows with the length of the conversation. On a long document it can become significant, and it is often what limits in practice the size of the text you can process.

Quantisation, or how to make a model fit

Quantisation means storing each parameter with less precision: eight bits, four bits, sometimes fewer, instead of sixteen. The same seven-billion-parameter model, at four bits per parameter, fits in a little over three and a half gigabytes, plus context memory. That puts it within reach of many machines.

Quality suffers, but less than you might think, up to a certain threshold. At eight bits the loss is usually negligible. At four bits it is modest on many tasks. Below that it becomes clear. It is not a uniform rule: it all depends on the model and the task, and the only way to know is to try on your own cases. Quantised models are distributed in suitable formats, the most widespread for local inference being GGUF.

Speed depends on bandwidth

To produce a word, the processor has to reread most of the model’s parameters. Generation speed is therefore limited less by computing power than by how fast memory can supply that data. A useful rule of thumb: the bigger the model and the slower the memory, the slower the generation.

This explains why computers with fast unified memory, such as Macs with Apple silicon, do well with these models, and why a small quantised model is often more pleasant to use than a large model that saturates the machine.

Tools to try

You no longer need to compile anything by hand. A few commonly used tools:

• llama.cpp, the reference inference engine in C++, which popularised the GGUF format and runs on CPU as well as GPU;
• Ollama, which downloads, manages and serves models with a command-line interface and a local API;
• LM Studio, an application with a graphical interface;
• MLX, Apple’s library for its chips, which takes advantage of their architecture.

Several of these tools expose an API compatible with the best-known online services. Concretely, an application written to call a remote provider can often be pointed at the local model by changing an address, which makes comparisons much easier.

What small models do well

A small model is not a large model in miniature, it has clear strengths.

• Classifying: sorting messages, detecting a subject, a language, a tone.
• Extracting: pulling structured fields out of a text, such as a date, an amount or a name.
• Rewording and summarising short texts.
• Routing: deciding which step of a process should run, or which larger model to call.
• Computing embeddings, for semantic search, which small specialised models do very well.
• Completing simple code inside an editor.

What they do badly

The same models fall short elsewhere: multi-step reasoning, long instructions with many constraints, rare factual knowledge, where they invent more readily than a large model. A small model that answers confidently on a specialised subject has a good chance of being wrong, and nothing in the tone of its answer says so.

The practical consequence is to give it bounded, verifiable tasks, where you can check the output with code or compare it to a list of permitted values, rather than open questions.

Specialisation, the hidden card

A small model can be fine-tuned on a narrow task, for example with the LoRA technique, which trains only a small number of additional parameters. On a precise, well-delimited task, such as sorting your tickets according to your own grid, a small specialised model can match a much larger model used as is, for a fraction of the cost. It takes good-quality examples, a few hundred at least, and a separate test set to check the result.

Why bother

The reasons to run a model at home are not only about fashion.

• Privacy: the data never leaves the machine.
• Availability: it works without a network, on a train, at an isolated site.
• Zero marginal cost: once the model is in place, each extra request costs only electricity.
• Stability: the model changes only when you decide, which makes results reproducible.
• Predictable latency, with no dependence on a third-party service’s load.

How to start, concretely

1. Pick a precise, repetitive task from your work: classifying, extracting or rewording.
2. Gather twenty to fifty real examples, with the answer you expect.
3. Install one of the tools above and download a model of three to eight billion parameters, quantised to four or eight bits.
4. Run it over your examples and look at the errors, not only the success rate.
5. If that is not enough, try a larger model, or another of the same size: the gap between models of comparable size is sometimes considerable.

The right model is the smallest one that succeeds at your task, measured on your data.

What you gain from this exercise goes beyond the question of local AI. You learn to write a task in a verifiable way, to build a set of examples and to measure. That is exactly the discipline that also serves when you go back to a large remote model, because you then know what you are asking of it and what you expect from it.