RAG vs fine-tuning: how to choose

What each approach actually changes

• Prompting changes the instructions and examples the model sees on each request. Zero training, instant iteration, and modern models follow detailed instructions well. Limited by context size and by how consistently the model follows instructions at scale.
• RAG (retrieval-augmented generation) changes the information in context at request time by searching your data and inserting the relevant passages. The model’s weights don’t change. Knowledge updates are as fast as re-indexing a document, and answers can cite sources.
• Tools (function calling) let the model fetch live data or take actions through your APIs – order status, account balance, creating a record. For anything that lives in a database, a tool beats both RAG and fine-tuning.
• Fine-tuning changes the model’s weights by training it further on examples of inputs and desired outputs. It’s good at teaching consistent formats, tone, domain-specific labels, and compressing a long prompt into the model so you can use a smaller, cheaper one. It’s a poor way to add facts: the model learns patterns from the training data, not a reliable, updatable store of knowledge, and it can’t tell you where an answer came from.

The decision order I use

1. Write the prompt properly first. Clear instructions, a handful of good examples, a structured output schema. Build an evaluation set of real cases and measure. Many “we need fine-tuning” problems end here.
2. If it fails because it lacks information, add retrieval (for documents) or tools (for live, structured data). Measure again.
3. If it fails because of behavior – inconsistent format, wrong tone, a classification boundary it can’t learn from a few examples – and you have hundreds or thousands of good examples, consider fine-tuning.
4. If it works but costs too much or is too slow at your volume, fine-tune a smaller model to match the larger one on that narrow task, using the large model’s good outputs (checked) as training data.
5. Combine as needed. A fine-tuned model that follows your answer format, fed by retrieval for facts, is a common and sensible end state.

Every step depends on the evaluation set from step one. Without it you can’t tell whether fine-tuning helped, or whether a later base model has made it unnecessary. See LLM evaluation.

When RAG is the right choice

• The answers depend on your private documents: help center, policies, contracts, internal wikis, tickets.
• The information changes – weekly, daily or per customer.
• Users need citations to trust or verify the answer, or you need an audit trail.
• Different users may see different content, so retrieval must respect permissions. You can filter what’s retrieved per user; you can’t filter what’s baked into weights.
• You want to delete information on request. Removing a document from an index is easy; removing it from a fine-tuned model means retraining.

The cost of RAG is mostly engineering and operations: ingestion pipelines, chunking, hybrid search, reranking, permission sync and evaluation. I go through the components in RAG architecture.

When fine-tuning an LLM is the right choice

• You need a specific output format or style applied consistently across high volume, and prompting gets it right most of the time but not reliably enough.
• The task is a narrow classification or extraction problem with plenty of labeled examples – routing tickets into your taxonomy, tagging documents with internal codes.
• You want to move a task to a smaller model for cost or latency, and you have data showing the big model’s outputs are good.
• Your prompt has grown huge with examples and edge-case rules, and those tokens are a large share of every request.

Methods range from full fine-tuning to parameter-efficient techniques like LoRA, which train a small set of additional weights and are what most open-weight workflows and many hosted fine-tuning services use. Hosted options are narrowing – OpenAI closed self-serve fine-tuning to new customers in 2026 and stops new jobs for existing ones in January 2027 – so check what your provider still offers before planning around it. The method matters less than the data: a few hundred carefully reviewed examples beat thousands of noisy ones.

Cost and maintenance trade-offs

• Prompting: cheapest to build and change. Cost scales with prompt length on every request; prompt caching softens that.
• RAG: moderate build cost, ongoing cost in ingestion, storage and keeping content clean. Each request carries retrieved context, so per-request token cost is higher than a bare prompt. Updating knowledge is cheap.
• Fine-tuning: cost of building and maintaining a training dataset, training runs, evaluation and hosting or premium inference rates for the tuned model. Every change to the desired behavior means a new dataset version and a new run. When the provider retires the base model, you retrain on a new one and re-evaluate.
• Model churn: base models improve quickly. A fine-tune that beat the general model last year may lose to this year’s general model with a decent prompt. Re-test before renewing the investment.

For the per-token side of these trade-offs, see OpenAI API cost.

Common mistakes

• Fine-tuning to teach facts, then finding the model blends them, misremembers them and can’t be updated without retraining.
• Fine-tuning before writing a serious prompt, so the baseline was never measured.
• Training on unreviewed data, which teaches the model your inconsistencies.
• Using RAG for questions a database answers better. “What’s my balance” is a tool call; “how many orders last month” is a query.
• Skipping evaluation and judging the fine-tune by a few hand-picked examples.

Fine-tuning vs RAG at a glance

• Adds new or changing knowledge: RAG yes; fine-tuning poorly.
• Teaches format, tone or narrow skills: fine-tuning yes; RAG no.
• Cites sources: RAG yes; fine-tuning no.
• Respects per-user permissions: RAG yes, with filtering; fine-tuning no.
• Time to update: RAG minutes; fine-tuning a new training cycle.
• Upfront data needed: RAG your documents; fine-tuning curated input–output examples.
• Can reduce per-request cost: fine-tuning yes, by enabling a smaller model; RAG usually adds tokens.

If you’re not sure which problem you have, that’s usually the first thing I work out in an AI consulting engagement – by building the eval set and the simplest working version, not by writing a strategy document. When the answer is retrieval, I build it as RAG development work.

Related

• RAG architecture for production systems
• RAG development services, measured on retrieval quality
• AI consulting services for software companies
• LLM evaluation for product features
• OpenAI API cost: estimate it, then cut it
• An AI knowledge base your team can actually trust