Local LLM vs the OpenAI API: What It Actually Costs to Self-Host

Almost every version of this comparison you will read is rigged, usually without meaning to be. It takes the electricity cost of running a model at home, compares it to the list price of a frontier API, and concludes that local is a hundred times cheaper. Both halves of that are wrong: the local figure leaves out the hardware, and the two sides are not the same product.

Here is the comparison done properly, with the arithmetic shown.

A realistic self-host: a used RTX 3090 at $900, running a 14B model at Q4_K_M. That is 24 GB at 936 GB/s, which puts it at about 95 tokens per second on that model. Electricity at $0.18 per kWh, 350W under load, and a four-year life on the card.

Local has two costs, and the second is the one that gets dropped:

• ▸Electricity while generating — about $0.18 per million tokens. This is the number everyone quotes.
• ▸The card itself, spread over its useful life — which depends entirely on how hard you work it.

At 10% utilisation, meaning the GPU is actually generating for about two and a half hours a day, that card produces roughly 300 million tokens a year and the all-in cost is about $0.93 per million tokens. Five times the electricity-only figure. At 2% utilisation it is $3.94. At 50% it falls to $0.33.

This is where the comparison usually cheats, because API prices span two orders of magnitude depending on which model you pick:

• ▸Cheap hosted small models: roughly $0.30 per million tokens blended. These are priced close to the metal.
• ▸Mid-tier: around $3 in and $15 out, which at a typical four-to-one input ratio blends to roughly $5.40 per million.
• ▸Frontier, such as GPT-6 Astra at $10 in and $50 out: roughly $18 per million blended.

Only against the expensive end, and only at volume. Break-even on that $900 card, after subtracting the electricity you still pay:

• ▸Against cheap hosted models at $0.30 per million — the card saves about $0.12 per million and needs 7.8 billion tokens to pay for itself. At 50,000 tokens a day that is over four centuries. In practice: never.
• ▸Against mid-tier at $5.40 per million — 172 million tokens. At 500,000 a day, about 345 days.
• ▸Against frontier at $18 per million — 50 million tokens. At 500,000 a day, about 101 days.

Read the first line again, because it is the one that matters most and the one nobody prints. If the thing you would otherwise use is a cheap hosted small model, buying hardware to replace it will not pay for itself in the life of the card. Those models are already priced near the cost of serving them.

The frontier row above is the one that makes local look good, and it is also the one that is not a like-for-like swap. A 14B model running on your desk is not a substitute for a frontier model. It is a different, weaker product that happens to be cheaper.

If you replace frontier API calls with a local 14B and your task still works, the honest conclusion is that you were overpaying for capability you did not need — and the cheapest fix was probably a cheaper hosted model, not a GPU. If the task stops working, you have not saved anything at all.

The fair comparison is a local model against a hosted model of similar capability. On that basis the API wins on cost almost every time, because the provider is amortising hardware across thousands of users and you are amortising it across one.

Because cost is not why local wins, and treating it as the argument is what makes the argument lose.

• ▸Privacy. Nothing leaves the machine. For client work, health data, unreleased code or anything under an NDA, this is not a preference — it is the requirement, and it makes the cost question irrelevant.
• ▸No metering. You can throw a hundred thousand tokens at an idea without watching a counter, which changes how you work more than the money does.
• ▸Offline and always available. No rate limits, no outages, no queue at peak.
• ▸Permanence. A model on your disk cannot be deprecated, price-changed or have its behaviour altered underneath you.
• ▸Latency. A small local model answers faster than a network round trip to a large one, which matters for autocomplete and interactive tools.

Those are real and they are worth money. They are just not the same argument as dollars per million tokens, and mixing the two produces the breathless posts that get this wrong.

• ▸Replacing a cheap hosted small model to save money: do not. The maths does not work.
• ▸Replacing frontier calls at serious volume, with a task a smaller model genuinely handles: local pays back in months, but try a cheaper hosted model first.
• ▸Privacy, offline use or unmetered experimentation: run locally and stop doing cost arithmetic, because those benefits are the reason.
• ▸Uncertain: use the API until it hurts. The bill is the signal, and hardware is easier to justify with a number in front of you.