Small models, big effects: AI that runs on your own machine
A model with a few billion parameters fits in the memory of an ordinary laptop once quantised. What it does well, what it does badly, and how to try it without installing anything exotic.
Category
A model with a few billion parameters fits in the memory of an ordinary laptop once quantised. What it does well, what it does badly, and how to try it without installing anything exotic.
Host a model yourself or pay for an API: the right answer depends on five criteria you can examine one by one. Real cost, privacy, latency, licence and dependency, without dogma.
The word “agent” is used to sell everything. Underneath sits a fairly simple loop: a model picks an action, a program runs it, a result comes back. What matters is the limits you put around it.