When a team wants an AI model to behave a certain way, they reach for one of three levers: prompting, retrieval-augmented generation (RAG), or fine-tuning. The three are often discussed as competing philosophies, which leads teams to over-invest in the most complex option when a simpler one would have worked. Choosing wrong is expensive in time, money, and credibility. Here is a framework that makes the decision obvious.
The three levers
Each technique changes a different part of the system, and that distinction is the whole game.
- Prompting changes the instructions you give the model. You are telling it what to do and how to do it.
- RAG changes the information the model can see. You retrieve relevant documents and put them in front of the model at the moment it answers.
- Fine-tuning changes the model itself. You train it on examples so the desired behavior is baked into its weights.
Almost every "should we fine-tune?" conversation gets clearer once you name which of these three problems you actually have.
Start with prompting
Prompting is the fastest and cheapest lever, and it solves far more than most teams expect. A clear, well-structured prompt with a few good examples can get a capable model to perform a surprising range of tasks reliably. Because there is no training run and no infrastructure to maintain, you can iterate in minutes and roll back instantly.
The rule of thumb: never reach for a heavier technique until you have genuinely exhausted prompting. Many fine-tuning projects exist only because nobody spent a focused afternoon improving the prompt first.
Reach for RAG when knowledge is the problem
If your model gives confident but wrong answers because it simply does not know your business, the problem is knowledge, not behavior. That is what retrieval is for. RAG connects the model to your private, current, or proprietary information, such as product documentation, policies, past tickets, or contracts, and supplies the relevant pieces at answer time.
RAG has two big advantages for the enterprise. It keeps your data current, because you update the knowledge base rather than retraining a model, and it is auditable, because you can show exactly which source a given answer came from. For most "answer questions about our stuff" use cases, RAG is the right first heavy investment, not fine-tuning.
Fine-tune when behavior is the problem
Fine-tuning earns its keep when you need the model to behave a particular way consistently, and prompting cannot get you there reliably. Good candidates include a very specific output format every single time, a distinctive brand voice, a narrow classification task at high volume, or a specialized skill the base model handles inconsistently.
Fine-tuning can also reduce cost and latency: once a behavior is trained in, you often need shorter prompts and can sometimes use a smaller model. The trade-off is real, though. You need quality training data, a place to host and version the model, and a plan to retrain as requirements drift. Fine-tuning is a commitment, not a quick fix.
They are not mutually exclusive
The most capable production systems usually combine all three. A common, powerful pattern is a fine-tuned model that reliably follows your format and tone, fed by RAG so its answers are grounded in current data, wrapped in a carefully engineered prompt that sets the task and the rules. Thinking of these as a stack rather than a choice prevents the false dilemma that derails so many projects.
A worked example
Imagine you want an assistant that answers customer questions about your products in your brand voice. Walk the framework. The assistant needs to know your current catalog, pricing, and policies, which is a knowledge problem, so RAG is essential and should come first. It also needs to sound consistently like your brand and always end with a specific next step, which is a behavior problem that prompting can usually handle at first and fine-tuning can lock in later if consistency slips at volume. And it needs firm rules about what it must never do, such as promising discounts, which is an instruction problem solved in the prompt and reinforced with guardrails. One use case, three levers, each applied to the part of the problem it actually fits.
Notice what the framework prevented: jumping straight to fine-tuning a model on support transcripts, an expensive project that would not have fixed the real gap, which was that the model could not see current pricing. The discipline of naming the problem type first saves both money and months.
A decision checklist
When a new use case lands on your desk, walk through these questions in order:
- Is this an instruction problem? Try prompting first. Improve the prompt, add examples, and measure.
- Is this a knowledge problem? The model lacks your data. Add RAG before anything heavier.
- Is this a behavior problem? You need consistency prompting cannot deliver. Now fine-tuning is justified.
- Is it more than one? Combine techniques deliberately rather than defaulting to the most complex.
The cost and maintenance reality
Each lever carries a different ongoing burden, and that should weigh heavily in the decision. Prompting is essentially free to maintain. RAG requires you to keep a knowledge base fresh and a retrieval pipeline healthy. Fine-tuning requires data pipelines, training runs, model hosting, and periodic retraining as the world changes. The right question is not only "which performs best in a test?" but "which can my team realistically operate for the next two years?"
Revisit the decision as you scale
The right answer changes with volume and maturity. A use case that starts with prompting and RAG may later justify fine-tuning once it runs at high volume and the savings from shorter, trained-in prompts become material. Conversely, a fine-tuned model can become a liability if requirements shift faster than you can retrain it. Treat the choice as a decision you revisit on a schedule, not a one-time architectural commitment, and let your evaluation numbers rather than intuition tell you when it is time to move to a heavier or lighter technique.
Prompting changes the instructions. RAG changes the knowledge. Fine-tuning changes the model. Name the problem first, and the right technique picks itself.
