Open models or closed APIs: how to choose

The “open models versus closed APIs” debate quickly turns into a quarrel of faiths. On one side, people for whom anything not hosted in-house is a data leak waiting to happen. On the other, people for whom self-hosting is a waste of time next to services that work out of the box. Both camps are right about part of the problem, and wrong to think it only has one part.

It is better to treat the question like any other architecture choice: a few criteria, some trade-offs, and a decision that is allowed to change.

What “open” means, and what it does not

The word is misleading. Most so-called open models are really open-weights models: you can download the parameter file and run it yourself. The training code and the data used are not always published, sometimes not at all. So it is not always open source in the sense we mean for software.

Above all, the licence varies a great deal. Some models are released under well-known permissive licences such as Apache 2.0 or MIT. Others come under custom licences that allow commercial use but set conditions: a user threshold, a ban on using the outputs to train another model, an acceptable-use policy to respect. Reading the licence before building a product on it is a habit to acquire, not a formality.

The five criteria

1. Real cost

An API is paid by usage, generally by the volume of text read and produced. The cost is proportional to consumption, with no upfront investment, and it is easy to estimate on a prototype. A self-hosted model is paid for in hardware or GPU rental, in operations time and in monitoring, whether you use it or not.

The consequence is well known: at low or very irregular volume, the API is almost always cheaper. At high, steady volume, with well-filled machines, self-hosting can become attractive. The trap is comparing the price per token on one side with the price of the GPU on the other while forgetting the real utilisation rate, the time of the people maintaining the system and the cost of an outage.

2. Privacy

With an API, your data leaves your perimeter to be processed by a third party. Serious providers offer contractual commitments: no training on your data, limited retention, hosting in a given region. That is enough for many use cases, and not enough for others, for example health data or trade secrets under strict rules.

Self-hosting keeps everything with you, which simplifies compliance on paper. It also moves the responsibility: it is up to you to secure access, to manage the logs (which contain the prompts) and to patch the server.

3. Latency and availability

A local model avoids a round trip over the Internet and does not depend on a third party’s load. That does not make it fast, though: a modest GPU runs a large model far more slowly than infrastructure sized for it. Conversely, an API can have latency spikes, rate limits and outages over which you have no control.

What matters is the latency your product needs. An overnight batch or an internal assistant tolerates a lot. Completion while someone is typing does not.

4. Capability

The most capable models on hard tasks, long reasoning, complex programming, subtle instructions, are often closed models. The gap with open models varies by task and moves quickly, in both directions. For many everyday uses, classifying, extracting, summarising, rewording, mid-sized open models are plenty.

The only way to know is to test on your data. A general leaderboard published somewhere does not tell you how a model behaves on your documents, in your language, with your vocabulary.

5. Dependency

An API can change without asking you: a model is retired, its behaviour shifts with a new version, prices and terms move. That breaks products that relied on a precise behaviour. A downloaded open model no longer changes: you freeze it, test it, keep it as long as you like. That is a real advantage for reproducibility.

There is a dependency on the other side too: on an inference software stack, on a type of hardware, on a rare skill in the team.

A method that avoids lock-in

The decision is less often made on a principle than on a series of small questions. Here is a short version.

• Can the data being processed legally and contractually leave your premises?
• What monthly volume, and how regular, a year from now?
• What maximum latency does your product tolerate?
• Does a mid-sized model succeed at your task on a real sample?
• Who would run the self-hosted system, and what happens when that person is away?

In practice, the most robust strategy is almost always the same: hide the provider behind an interface. Your code calls a “generate”, “classify” or “compute an embedding” function, and a single place decides which model answers. Changing provider, or letting two coexist, becomes a configuration matter instead of a rewrite.

Choosing a model is a reversible decision, provided you made it reversible.

Hybrid approaches

Nothing forces you to pick a side for everything. A sensible organisation can send non-sensitive data to an API for hard tasks, keep confidential documents local with a smaller model, and use a local embedding model for indexing. Routing by data sensitivity or task difficulty often does better than a single choice.

Where to start

For a new project, prototyping with an API is almost always the shortest path: you learn what your product really asks of the model before investing in infrastructure. Once the task is understood, replay your tests with one or two open models and look at the gap. If it is acceptable, and if your volume or privacy constraints justify it, migrating is a well-defined project rather than a gamble.

And in every case, keep a small set of real examples to compare against. It is the only thing that makes the discussion concrete.