Fine-tune or prompt

'We're going to fine-tune a model on our data.' I hear it at almost every first conversation, said with the confidence of someone who has found the serious answer. In the vast majority of cases, it is the wrong reflex: more expensive to set up, slower to ship, and rarely better than what a good prompt would have produced in an afternoon. Fine-tuning retains an aura of seriousness it does not always deserve. Here is how I decide, every time, between fine-tuning and prompting.

Fine-tuning or prompting: what's actually the difference?

Prompting means writing good instructions for an already-trained model: you tell it what to do, how to do it, with a few examples, on every call. Fine-tuning means retraining that model on your own examples to change its behaviour at a deeper level. That is where the confusion starts: people picture fine-tuning as something that 'teaches the model their business', like a training course it remembers forever. In practice, prompting gets the same result nine times out of ten, without retraining anything and without tying up a team for weeks. So before taking on the heavy work, I check two things: that the need is real, and that prompting has genuinely hit its limits, not just that it was done sloppily.

What does prompting already handle, with no technical debt?

The vast majority of needs come down to good instructions and a few well-chosen examples. All of it is steered from the prompt, immediately, with no training and no corpus to assemble: it is adjustable on the spot, versioned like code, and it creates no technical debt. When a case proves stubborn, you refine the guidance and add two or three examples, not a training pipeline to maintain.

• Output format: JSON, a table, a fixed template, obtained through the instruction and one or two examples.
• Tone and style: formal, direct, commercial, set in a single line of guidance.
• Business rules: what to do, what to refuse, the edge cases, spelled out in black and white.
• Up-to-date knowledge: injected into the prompt when needed, rather than baked into the model.

In other words, before you even touch the model's weights, you already have a lot of room just by writing better. Most of the projects I take over had not exhausted that room: they had jumped straight to the idea of fine-tuning, mistaking 'it resists' for 'prompting can't do it'.

Why does fine-tuning cost more than it looks?

Because the price of training is the visible part, and the cheapest. The real cost is elsewhere. First you have to gather and clean hundreds, sometimes thousands of quality examples, which assumes your data is ready, rarely the case at the start. Then a fine-tuned model has to be maintained: every time the provider ships a stronger version, your fine-tune stays stuck on the old one, and you have to redo everything to benefit from the progress. It is that recurring cost, not the initial bill, that sinks the deal. Where a prompt is fixed in five minutes, a fine-tune has to be retrained, retested and redeployed every single time.

A market signal came along to confirm that arithmetic. In May 2026, OpenAI announced it was winding down its self-serve fine-tuning API, explaining that recent models follow instructions and formats far better, and that prompt-based approaches had become cheaper and faster. New training runs stop in early 2027, and inference on already fine-tuned models stays available. When the vendor selling the option closes the counter and gives that reason, it beats any argument I could line up.

• Data preparation: gathering, cleaning and labelling enough quality examples, the heaviest and most underestimated task.
• Maintenance: with every new model version, your fine-tune ages and has to be redone to stay competitive.
• Rigidity: fixing a behaviour takes a retraining run, not a changed line of instruction.
• Evaluation: without a solid test set, you don't even know whether the fine-tune beats the prompt it replaces.

Wanting 'a model that knows your business': is that really fine-tuning?

Most of the time, no. When someone asks me to fine-tune 'so the model knows our products, our procedures, our contracts', they are describing a knowledge need, not a behaviour need. But fine-tuning changes how the model answers, not what it knows about your documents. To give it access to your up-to-date information, the right approach is almost always to fetch it at question time and hand it to the model in the context: that is the principle of RAG. I laid out this choice in RAG or fine-tuning, because confusing the two makes you spend a lot to solve the wrong problem. The rule fits in one line: fine-tuning for the form, RAG for the facts.

When is fine-tuning genuinely worth it?

In a minority of cases, when prompting hits a ceiling despite every effort and the numbers themselves call for something else. Concretely, three situations justify it.

• A very specific style or format, required at very high volume, that no instruction stabilises well enough.
• A latency or a cost to compress on a task repeated millions of times, where a small fine-tuned model beats a large prompted one. Before getting there, I check the other levers to cut the bill.
• A highly specialised domain where prompting truly plateaus, with examples to prove it, not just an impression.

What these three cases share is volume. Fine-tuning only pays off at scale, when the per-call gain, multiplied by a very large number of calls, finally beats the cost of preparation and maintenance. Below that threshold, the math does not hold, and the prompt stays the better choice, by a wide margin.

How do you decide without getting it wrong?

By always following the same order, and refusing to skip steps. First, prompt seriously: clean instructions, representative examples, then measurement on real cases, not on three examples picked by hand. Only then, if a ceiling shows up in black and white, consider fine-tuning and compare it to the prompt on exactly the same tests. Fine-tuning from the start means tying yourself to a specific model and version before you even know whether prompting was enough, and paying to find out. The reverse order costs a lot and teaches nothing.

Fine-tuning is not proof that you take AI seriously. More often, it is proof that you didn't try hard enough to prompt.

Before launching a fine-tuning, ask yourself one simple question: have you really pushed prompting to its ceiling, with numbers to show it? Nine times out of ten, the answer is no, and the saving is immediate. If you are unsure about your specific case, or want to check that a fine-tune already in place still earns its keep, let's talk: a first conversation is often enough to decide, with no commitment.