Fine-Tuning, RAG, or Prompting: Choosing the Right Approach
A decision framework for teams weighing prompt engineering, retrieval-augmented generation, and fine-tuning — what each technique actually fixes, what it costs to run, and how to combine them.
Start from the failure, not the technique
Teams usually arrive at this question backwards: they pick fine-tuning or RAG first, then look for a problem it solves. The productive order is the reverse. Look at a sample of wrong outputs and classify what went wrong.
- The model lacked a fact. It did not know your pricing, your schema, last week's policy change. This is a knowledge problem.
- The model knew enough but behaved wrongly. Wrong format, wrong tone, ignored constraints, inconsistent structure. This is a behaviour problem.
- The model was not asked clearly. Ambiguous instruction, missing context, contradictory requirements. This is a specification problem.
Specification problems are fixed by prompting. Knowledge problems are fixed by retrieval. Behaviour problems that survive good prompting and good examples are the narrow case where fine-tuning earns its keep.
What each approach actually does
Prompting
Instructions, examples, and output schemas, all resolved at request time. Nothing to train, nothing to host; iteration takes minutes and rollback is a revert.
Its ceiling is the context window and the model's own capabilities. Long instructions cost tokens on every single call, and very long prompts get expensive and harder to keep coherent.
Choose it as the default. Almost every project should exhaust prompting before considering anything heavier.
Retrieval-augmented generation
A retrieval step fetches relevant content and injects it into the prompt. This is how you give a model knowledge it never saw in training, with citations and without retraining.
Costs are architectural rather than computational: an ingestion pipeline, an index to keep fresh, permission filtering, and an extra hop of latency. Quality becomes an information-retrieval problem — if retrieval misses the passage, no prompt recovers it.
Choose it when answers depend on your own or fast-changing content, when users need sources, or when access control must stay in your systems.
Fine-tuning
Additional training that adjusts model weights toward your examples. It is good at shaping how a model responds: adopting a house format, a domain's terminology, or a consistent classification boundary. It can also let a smaller, cheaper model match a larger one on a narrow task — a real cost lever at high volume.
What it does not do reliably is install facts. Knowledge encoded in weights cannot be updated when the document changes, cannot be cited, and cannot respect per-user permissions.
The costs are ongoing: a curated, labelled dataset, a training and evaluation loop, versioned artefacts, and a migration each time you want to move to a newer base model.
Choose it when you have hundreds to thousands of high-quality examples, a stable and narrow task, prompting has plateaued, and either volume makes a smaller model economically attractive or the required behaviour is genuinely hard to express in words.
Cost and maintenance at a glance
| Prompting | RAG | Fine-tuning | |
|---|---|---|---|
| Time to first version | Hours | Days | Weeks |
| Updating knowledge | Edit prompt | Reindex document | Retrain |
| Per-request cost | Prompt tokens | Prompt + retrieval | Often lower (smaller model) |
| Can cite sources | No | Yes | No |
| Main risk | Context limits | Retrieval misses | Stale weights, drift |
The column that surprises people is maintenance. Prompts and indexes are cheap to change; a fine-tune is a small ML project with a lifecycle, and it must be redone whenever the base model moves on.
They compose — and usually should
These are layers, not rivals. A mature system often looks like: a fine-tuned small model for a high-volume classification step, retrieval for the knowledge-heavy answering step, and carefully versioned prompts with enforced output schemas throughout.
A workable progression:
- Prompt well. Clear instructions, a strict output schema, a few honest examples. Measure on a real evaluation set.
- Add retrieval when the errors are missing facts. Measure retrieval recall separately from answer quality.
- Consider fine-tuning only when the remaining errors are stubborn behaviour on a stable task, and you have the labelled data to support it.
Before you commit
Ask three questions and be honest about the answers.
- How often does the underlying knowledge change? Weekly changes and fine-tuning are a poor match.
- Do you have evaluation data? Without a measurable baseline, you cannot tell whether the expensive option helped, and both RAG and fine-tuning add moving parts that need proving.
- Who maintains it in a year? Every layer added is a system somebody must keep running when the original authors have moved on.
The default answer for most teams is: prompt carefully, retrieve when knowledge is the gap, and reach for fine-tuning last — deliberately, with data in hand, and for a specific measured reason.