When to Fine-tune vs Prompt-engineer
Teams reach for fine-tuning far too early. It feels like the "serious" option, but it's usually the last lever you should pull, not the first. Here's the order I actually recommend, and how to know when to move to the next step.
Start with prompting
Before anything else, exhaust prompt engineering: clear instructions, few-shot examples, structured output, and decomposition into smaller steps. A surprising share of "we need a custom model" problems dissolve with a better prompt and a couple of well-chosen examples. It's free, instant to iterate, and easy to change.
Add RAG when the gap is knowledge
If the model is capable but doesn't know your data — internal docs, product catalog, recent events — the fix is retrieval, not fine-tuning. RAG injects fresh, citable knowledge at query time and updates by re-indexing. Fine-tuning bakes knowledge in at a snapshot and goes stale.
Rule: fine-tune to change behavior, retrieve to change knowledge.
Fine-tune when the gap is behavior
Fine-tuning earns its cost when you need consistent style, format, or domain-specific reasoning that prompting can't reliably enforce at scale. Good signals:
- You have a few hundred to a few thousand high-quality labeled examples.
- The task is narrow and repeated at high volume (so a smaller fine-tuned model beats a large general one on cost/latency).
- Prompt length is ballooning with examples just to hold quality — fine-tuning can move that into weights.
The costs people forget
- Data curation is the real work — garbage examples produce a confidently wrong model.
- Maintenance — every base-model upgrade may require re-tuning.
- Evaluation — without a held-out benchmark you can't tell if a tune helped or hurt.
A simple decision path
- Can a better prompt fix it? → Do that.
- Is it missing knowledge? → RAG.
- Is it inconsistent behavior at scale, with data to teach it? → Fine-tune.
- Still stuck? → Combine: fine-tune for behavior, RAG for knowledge.
Most production systems I've built land on prompting + RAG, with fine-tuning reserved for the few high-volume, behavior-critical paths where it clearly pays for itself.