Fine-tuning vs RAG vs prompting: a decision framework

Every team building an LLM-powered feature eventually faces the same three-way fork: should we prompt-engineer our way to the behavior we need, pull in relevant context via retrieval, or bake the knowledge into the model weights through fine-tuning? The answer is rarely obvious, and picking the wrong approach wastes weeks of engineering time, model-provider budget, or both.

This post gives you a decision framework grounded in trade-offs — cost, latency, maintenance burden, and where each approach genuinely breaks down. We’ll also cover how the three compose, because production systems usually end up using more than one.

What each approach actually does

Before trade-offs, a precise framing of the three techniques:

Prompt engineering means giving the model all the information and behavioral instructions it needs inside the context window — system prompts, few-shot examples, chain-of-thought instructions, output format specifications. The model weights never change. The only artifact is text.

Retrieval-augmented generation (RAG) splits the problem: a retrieval system finds the passages most relevant to the user’s query, those passages are injected into the context, and then the base model generates a response grounded in that retrieved material. The model weights don’t change. The artifact is an index plus a retrieval pipeline.

Fine-tuning updates the model’s weights on a curated dataset so the desired behavior is encoded directly in the model rather than induced at inference time. The artifact is a new model checkpoint. The base model’s context window is still available — fine-tuning and prompting are not mutually exclusive.

The decision tree

Start with prompt engineering

Prompt engineering is always the right first experiment. It is the cheapest approach with the shortest iteration loop. If a well-crafted system prompt and a handful of few-shot examples get you 80% of the way to your quality target in a day, that is your answer for now.

Prompt engineering works well when:

• The behavior you want is expressible as instructions (“respond only in JSON”, “always cite sources”, “refuse medical advice”)
• Your knowledge base is small enough to fit in context (under ~50,000 tokens in practice, depending on model)
• Requirements are likely to change frequently — a prompt is edited in minutes; a fine-tune takes hours and dollars

Prompt engineering breaks down when:

• Knowledge is too large to fit in context and changes frequently
• The model systematically ignores instructions despite rephrasing — this is a distributional problem, not a prompting problem
• Latency is unacceptable and you’re paying for large prompt tokens on every request
• You need consistent output format or style that the base model resists through prompting alone

Move to RAG when knowledge is the bottleneck

If the gap between your model’s behavior and your target is primarily an information gap — the model gives confident wrong answers because it simply doesn’t have the right facts — RAG is the correct lever to pull.

RAG excels when:

• Your knowledge changes frequently (product documentation, internal policies, support KB, code repositories)
• The knowledge corpus is large relative to the context window
• You need attribution and citation — retrieved chunks give you direct provenance
• You want to swap, update, or extend knowledge without retraining

RAG does not solve:

• Style and format problems — if you want the model to respond as a specific persona or in a particular structure, retrieval doesn’t help
• Reasoning capability gaps — if the base model can’t reason over retrieved text correctly, more or better retrieval won’t fix it
• Latency in latency-critical paths — a retrieval round-trip adds 50–200ms before the first token in typical vector-search deployments
• Hallucination completely — models can still confabulate even with retrieved context, especially if the retrieved chunks are partially relevant

One underappreciated failure mode of RAG is precision degradation. As your chunk count grows and your query distribution widens, the retriever starts returning chunks that are topically adjacent but not actually useful. The model, presented with marginally relevant context, has a tendency to over-anchor on it. Monitoring retrieval precision — not just recall — is essential.

Fine-tune when the behavior is the bottleneck

Fine-tuning is the right tool when the behavior gap is distributional: the model you’re using has never seen enough examples of what you need to do that prompting can elicit it reliably. Classic signals that fine-tuning is warranted:

• Style and tone are hard to prompt into existence. If your brand voice requires consistent patterns that fight the base model’s defaults, encode them in weights.
• Domain jargon and entity recognition. Medical, legal, and technical domains often have entity types that base models handle inconsistently. Fine-tuning on domain text normalizes this.
• Structured output reliability. If JSON or XML mode with a schema still produces malformed output at a rate you can’t tolerate, a fine-tune on schema-conformant examples often fixes it.
• Latency and cost via distillation. A GPT-4-class model producing high-quality output can be used to generate a supervised dataset. Fine-tuning a smaller model on that dataset lets you run a cheaper, faster model without rebuilding the pipeline.

Fine-tuning has real costs that teams routinely underestimate:

Cost dimension	Typical magnitude
Dataset curation	40–80 engineering-hours for a quality 1000-example set
Training job	$20–$500 per run depending on model and dataset size
Evaluation	Must build eval before fine-tune to know if it worked
Maintenance	Every model update potentially requires a new fine-tune
Hosting	Fine-tuned checkpoints need dedicated hosting if self-serving

The biggest hidden cost is maintenance. A fine-tuned model is a snapshot. The underlying behavior is frozen at the moment the training data was cut. When your requirements evolve — new product features, revised policies, expanded domain — you either retrain or accept drift. Teams that fine-tune and then treat the model as done are setting themselves up for a slow degradation that’s hard to attribute.

The cost/latency matrix

Here is a practical approximation for a mid-complexity feature across the three approaches, assuming a 70B-scale model via API:

Approach	Marginal cost per 1K requests	p50 first-token latency	Maintenance overhead
Prompt-only (4K tokens avg)	~$0.50–2.00	300–600ms	Low — edit a file
RAG + prompt (2K tokens avg)	~$0.30–1.20 + retrieval infra	400–800ms	Medium — index upkeep
Fine-tuned small model	~$0.05–0.20	100–300ms	High — retrain on change

These numbers are illustrative, not benchmarks — they shift dramatically with provider, model size, and prompt structure. The key pattern holds: fine-tuning smaller models wins on marginal cost at scale once you’ve absorbed the training and maintenance overhead. RAG’s primary cost driver is the base prompt tokens plus retrieval infrastructure.

How they compose

The real decision is not which one to use — it’s which combination. Production systems often need all three:

Prompt + RAG is the most common pairing. The system prompt sets the persona, format, and behavioral constraints; RAG fills in the domain knowledge. This is the correct starting point for most knowledge-intensive applications.

Prompt + fine-tune is common for style and format consistency. You fine-tune for behavior, then use the context window for dynamic per-request instructions that are too variable to encode in training data.

RAG + fine-tune is used when you need both domain knowledge and adapted behavior. For example, a medical coding assistant might fine-tune on clinical text for entity recognition, then use RAG to retrieve the specific billing policies and code definitions relevant to each case.

Prompt + RAG + fine-tune is the full stack. It is expensive to build and maintain, so it should be reserved for high-value, stable features where you’ve validated the incremental gain of each layer. Don’t compose all three speculatively.

Eval-first development

The framework above is only useful if you can measure quality. Before choosing an approach, build an evaluation set. It doesn’t need to be large — 100–200 representative examples with expected outputs or scoring criteria is enough to get started. Run every variant of your approach against the same eval. This makes the decision data-driven rather than intuition-driven, and it gives you a baseline to catch regressions as you iterate.

A common trap: teams optimize for the approach that performs best on a small eval set they built themselves. This is overfitting in disguise. Hold out a portion of your examples for final validation and don’t look at it during iteration.

When to reconsider your base model choice

The three-way decision assumes a fixed base model. That assumption is worth revisiting before you invest in fine-tuning. Swapping to a more capable base model is sometimes cheaper than fine-tuning a weaker one. Conversely, swapping to a smaller model and fine-tuning it is sometimes cheaper than running a larger model with RAG. Multi-provider routing infrastructure — where your gateway can direct different request types to different models — gives you the flexibility to run these comparisons in production without rebuilding your application layer.

Where ManyLayers fits

Teams using ManyLayers manage prompt templates and RAG pipelines in the Workspace module, route requests to fine-tuned checkpoints alongside hosted providers through Gateway, and deploy custom model weights via the Deploy module — all behind a single OpenAI-compatible endpoint. The value of this architecture is that the choice of prompt-only versus RAG versus fine-tune becomes a routing and configuration decision, not an application-code change. You can A/B test approaches in production, collect per-approach cost and latency data, and make the right decision with production evidence rather than dev-environment intuition.

The decision framework is: prompt first, retrieve when knowledge is the bottleneck, fine-tune when behavior is the bottleneck, and compose when you need both. Start with the cheapest experiment, build your eval set before you build your pipeline, and instrument everything so you know which lever to pull next.