RAG vs Fine-Tuning
Retrieval-Augmented Generation (RAG) is the practice of grounding a language model in external knowledge at query time, retrieving relevant passages from your corpus into the prompt. Fine-tuning is the practice of changing the model's weights on your own examples, baking format, tone, or task behaviour into the model itself.
Use RAG to give a model knowledge and fine-tuning to give it behaviour. That single distinction decides most enterprise cases. If the answer depends on facts that change, live in documents you already control, and need to be traceable to a source, retrieval is the right mechanism: you update the corpus and the system is current in the next request, and the answer can cite the passage it came from. If the problem is that the model formats output inconsistently, misses a domain convention, or performs a narrow task poorly no matter how you prompt it, fine-tuning is the right mechanism, because you are changing the behaviour rather than supplying missing facts. The two are not rivals and production systems frequently use both, but they fail differently: a retrieval system fails by fetching the wrong passage, and a fine-tuned model fails by confidently applying learned behaviour to a case it was never shown.
What is the difference between RAG and fine-tuning?
| Dimension | RAG | Fine-tuning |
|---|---|---|
| What it supplies | Knowledge, at query time | Behaviour, format, tone, narrow task skill |
| Update speed | Immediate, reindex the changed documents | Slow, requires a training run and promotion |
| Source of truth | Your documents, kept current outside the model | Baked into the weights at training time |
| Citations | Yes, answers can cite the retrieved passage | None natively |
| Access control | Enforceable per request, filtered against the caller's entitlements | Not enforceable, training data is absorbed |
| Right to erasure | Delete the document, reindex, purge caches | Requires retraining from a corrected dataset |
| Dominant failure | Wrong or missing passage retrieved | Learned behaviour misapplied to unseen cases |
| Cost profile | Retrieval, reranking, and longer prompts per request | Training run, evaluation, and checkpoint hosting |
| Evaluation | Recall at k, nDCG, faithfulness, citation accuracy | Task success against a held-out labelled set |
When should you use RAG?
RAG is the default for enterprise assistants and agents, for three reasons that have little to do with model quality.
The first is freshness. Policies, catalogs, contracts, runbooks, and tickets change weekly. A retrieval corpus tracks those changes through an ingestion pipeline; weights do not.
The second is access control. Retrieval lets you stamp access metadata onto each chunk at ingest and filter results against the caller's entitlements on every request, so an assistant answering a junior analyst cannot surface a document that analyst could not open directly. Once content is absorbed into weights, that boundary is gone. The same property covers deletion: honouring a removal request means deleting the document and reindexing, not retraining.
The third is verifiability. A grounded answer can cite the passage it used, and a citation validator can check those references against the identifiers actually retrieved. That is what makes an answer reviewable rather than merely plausible, and it is why retrieval sits underneath the governance controls described on what governed AI agents require.
Production retrieval is more than a vector search. Hybrid retrieval fuses dense vectors with lexical matching, a cross-encoder reranks the candidates, a relevance threshold drops weak ones instead of padding the context window, and a sufficiency check routes to a fallback or an abstention when the retrieved set does not support an answer. Chunking decides the ceiling: you cannot retrieve what chunking destroyed, so structure-aware splitting and metadata stamping at ingest matter more than the choice of index.
When should you fine-tune instead of using RAG?
Fine-tuning earns its cost in a narrower set of cases:
- Format and convention are the problem. The model has the knowledge but will not consistently produce the structure your downstream system parses, and prompt engineering has plateaued.
- A specialised task the base model does poorly. Classification against your taxonomy, extraction from an unusual document layout, or a domain register the base model does not hold.
- Latency or cost pressure at volume. A smaller tuned model that matches a larger prompted one on a narrow task can be cheaper to serve, if you are willing to operate the checkpoint.
- The behaviour is stable. Fine-tuning is worth doing when the target will not shift every quarter, because each shift means another run.
The preconditions are real. You need quality labelled examples, a held-out evaluation set defined before the run, reproducible dataset pinning so a run can be repeated, and a promotion gate that blocks a checkpoint which improves one metric while regressing another. Treat a checkpoint as a production dependency with an owner, a version, and a rollback path.
When should you combine them?
The strongest systems fine-tune for reliable behaviour and retrieve for current knowledge. The order matters: get retrieval working and evaluated first, because most quality complaints attributed to the model turn out to be retrieval problems, and a fine-tune layered on a bad corpus produces confident nonsense faster.
Retrieval is also the part teams most often get wrong at scale: grounding on the wrong corpus, no evaluation harness, cross-tenant leakage through a missing access filter, and agents writing to systems of record on the strength of an ungrounded answer. Getting the operational discipline right is the subject of LLMOps vs MLOps, and the identity controls that keep retrieval tenancy-isolated are covered in workload identity for AI agents and on AI agent security.
How Citadel helps
Citadel builds retrieval platforms grounded in your own corpora inside your tenancy: ingestion and chunking, hybrid retrieval with reranking, access filtering per request, groundedness and citation checks, and an evaluation harness that gates every change. Retrieval is a shared spine rather than a pipeline each team rebuilds, which is what keeps a large agent catalog consistent on access control and evaluation. See enterprise AI agents for how agents bind to it, and zero trust for AI for the surrounding posture.
Where to start
The Agent Assurance Assessment is a fixed-scope, fixed-price three-week engagement that inventories what is already running, produces an evidence pack mapped to NIST AI RMF and ISO/IEC 42001, and leaves one governed agent live on a real workflow. The Enterprise AI Agent Blueprint is a free governance guide covering identity, credentials, autonomy tiers, and audit. To scope a retrieval build, book a discovery call or read the enterprise consulting overview.