RAG vs Fine-Tuning: When to Use Each
Updated: Oct 1
UIT University 365 Institute of Technology
Series AI Engineering | Level Basic (Free)
Duration 25 minutes | Access Free
IT Engineering, AI and Applied AI, Data Science, Software Development, Digital Transformation
UNOP Sound (University 365 Neuroscience Oriented Pedagogy)
Take five minutes to prepare your brain. Play the isochronous tone track (40Hz gamma frequency) with your eyes closed. Gamma-frequency tones before a learning session raise attention and make the material easier to absorb.
[Audio player: UNOP Pre-Lecture Isochrone (40Hz, 5 minutes)]
In this Lecture
The Hook: Your Question, Answered
Your team needs a chatbot that answers customer questions using your company documentation. Someone says "let's fine-tune GPT on our docs." Someone else says "we should use RAG." Who is right?
In 2026, the answer is almost always: start with RAG, then add fine-tuning only for the specific behaviors RAG cannot fix. Fine-tuning teaches a model how to behave. RAG gives a model the facts it needs. Most teams confuse the two and fine-tune when they should retrieve.
The most common mistake in LLM application development is using fine-tuning to teach facts. Fine-tuned models go stale the moment your data changes. RAG systems update by replacing documents, no retraining required.
In the next 20 minutes, you will understand exactly when to use each approach, when to combine them, and when to use neither.
What Is RAG? Retrieval Augmented Generation Explained
RAG connects a language model to an external knowledge source at inference time. The model stays unchanged. Instead of baking knowledge into model weights, you store documents in a searchable index and retrieve relevant chunks at query time.
The RAG Pipeline
A modern RAG pipeline has four stages:
1. Indexing: Documents are chunked into passages (typically 400 to 600 tokens), embedded into vector representations, and stored in a vector database. Modern systems also index for keyword search (BM25) alongside vector search.
2. Retrieval: A user query is embedded and matched against the index. Hybrid retrieval combines dense vector similarity and sparse keyword matching to find the most relevant passages.
3. Re-ranking: A cross-encoder re-ranker (such as Cohere Rerank or BGE Reranker) re-orders the top 20 to 50 candidates by true relevance to the query. This is the single biggest quality lever in RAG, and the step most teams skip.
4. Generation: The top-ranked passages are inserted into the LLM prompt as context. The model generates an answer grounded in those passages, often with citations.
Why RAG Works
- Updates are instant: Replace a document in the index and the next query uses the new content. No retraining cycle.
- Sources are traceable: Every answer points to the specific passages that grounded it. This matters for compliance, audits, and user trust.
- Models are swappable: Move from GPT to Claude to Llama without redoing your retrieval pipeline. The knowledge lives in the index, not the weights.
- Cost is predictable: You pay for embedding API calls and vector database storage. No GPU training costs.
What Is Fine-Tuning? Adapting Model Weights
Fine-tuning modifies the model itself. You train the model on examples of inputs and desired outputs, adjusting its internal weights to produce responses that match your training data. The new behavior is baked into the model.
Types of Fine-Tuning
Full fine-tuning updates all model parameters. It requires significant compute (multiple GPUs, hours to days of training) and produces a large artifact. It is the most powerful form of fine-tuning but the hardest to maintain and roll back.
LoRA (Low-Rank Adaptation) trains a small adapter module (often less than 1% of model parameters) that sits on top of the frozen base model. LoRA is cheap, fast, reversible, and stackable. You can train multiple LoRA adapters for different tasks and swap them at inference time. For most fine-tuning use cases in 2026, LoRA is the right tool.
QLoRA extends LoRA with 4-bit quantization of the base model, reducing memory requirements further. This lets you fine-tune a 70B model on a single consumer GPU.
DPO (Direct Preference Optimization) and its variants (KTO, ORPO) have largely replaced classic RLHF in 2026. Instead of training a separate reward model and running PPO, you train directly on preference data: pairs of outputs where one is labeled better than the other. DPO is simpler, cheaper, and more stable than RLHF.
What Fine-Tuning Changes
Fine-tuning changes how the model behaves, not what it knows. A fine-tuned model produces outputs in the style, format, or tone of your training data. It does not learn new facts from fine-tuning data. If you fine-tune on your product documentation, the model may memorize some of it, but this knowledge goes stale the moment the documentation updates.
The Core Difference: Knowledge vs Behavior
This is the single most important distinction in this lecture. Get this right and you will avoid the most expensive mistake in LLM development.
RAG Handles Knowledge
Use RAG when the model needs information it did not see during training: your product catalog, internal policies, customer history, legal documents, real-time data. RAG retrieves the facts at query time, so the model always answers from current data.
Fine-Tuning Handles Behavior
Use fine-tuning when the model has the right facts but produces them in the wrong way: wrong tone, wrong format, wrong structure, wrong language register. Fine-tuning teaches the model to consistently produce outputs that match your desired style.
The Rule
RAG for facts. Fine-tuning for behavior. If you need the model to know something, retrieve it. If you need the model to do something differently, fine-tune it.
Teams that fine-tune to teach facts end up with models that are expensive to train, expensive to maintain, and stale the moment the data changes. Teams that use RAG for everything end up with systems that cannot hold a consistent tone or format. The best systems use both, each for what it does best.
When to Choose RAG: Five Scenarios
Scenario 1: Dynamic Knowledge
Your knowledge base changes frequently. Product prices update weekly. Policies change monthly. Support tickets arrive daily. Fine-tuning cannot keep up because each update requires a new training cycle. RAG updates by replacing documents in the index, which takes minutes.
Scenario 2: Source Citation Required
You need to show users or auditors where the answer came from. Fine-tuned models cannot point to a source document. RAG retrieves specific passages and can cite them directly. This is critical for legal, medical, financial, and compliance applications.
Scenario 3: Large Knowledge Base
Your corpus is larger than the model context window. Even with 1M-token windows, a large enterprise documentation set exceeds what fits in a single prompt. RAG retrieves only the relevant chunks, keeping the prompt size manageable.
Scenario 4: Multiple Knowledge Sources
You need to answer questions that span multiple data sources: internal docs, external APIs, databases, web pages. RAG can query multiple indices and combine results. Fine-tuning bakes one dataset into the weights and cannot mix sources at query time.
Scenario 5: Rapid Prototyping
You need a working system in days, not weeks. RAG pipelines can be built with off-the-shelf components (embedding model, vector database, LLM) in a few days. Fine-tuning requires data preparation, GPU provisioning, training, evaluation, and iteration cycles that take weeks.
When to Choose Fine-Tuning: Three Scenarios
Scenario 1: Style, Tone, and Format Consistency
Your model gets the facts right (from RAG or from its training data) but produces outputs in the wrong style. It sounds too generic when it should sound like your brand. It writes paragraphs when you need structured JSON. It uses casual language when you need formal clinical language. A few hundred to a few thousand curated examples in a LoRA adapter lock in the desired behavior.
Scenario 2: Distillation for Cost and Latency
A frontier model (GPT-5, Claude Sonnet) can already do your task well with the right prompt, but the API cost at production volume is too high. You fine-tune a small open-source model (7B to 13B parameters) on the frontier model outputs for your specific task. The tuned small model delivers near-frontier quality at roughly one-tenth the inference cost and a fraction of the latency. This is the strongest commercial case for fine-tuning in 2026.
Scenario 3: Specialized Domain Reasoning
Your task requires domain-specific reasoning that the base model does not handle well out of the box. Examples include medical report structuring, legal contract clause extraction, or financial document analysis. Fine-tuning on domain examples teaches the model the reasoning patterns specific to your field. Pair this with RAG for the actual facts, and you get both specialized reasoning and current knowledge.
The Hybrid Pattern: Best of Both Worlds
Most production systems that work well in 2026 use a hybrid approach. The pattern is straightforward:
Step 1: Build RAG First
Start with RAG. Prove the use case. Measure quality with an evaluation set. Collect data on where the system fails. This phase typically takes 1 to 3 weeks.
Step 2: Identify Residual Failures
After RAG is working, identify the failure modes that retrieval cannot fix: inconsistent formatting, wrong tone, JSON schema violations, domain reasoning gaps. These are behavioral problems, not knowledge problems.
Step 3: Fine-Tune for the Residuals
Fine-tune a smaller, cheaper base model to handle the behavioral residuals. Train a LoRA adapter on examples of the desired output format, tone, or reasoning pattern. Run the fine-tuned model inside the same RAG pipeline.
Step 4: Maintain Both Layers
The RAG layer handles knowledge updates by replacing documents. The fine-tuned adapter handles behavior. When a new base model comes out, you can retrain the adapter on the new base with a fraction of the original effort, and the RAG pipeline stays unchanged because it is model-agnostic.
This hybrid pattern stacks the strengths: live facts from retrieval, locked behavior from fine-tuning, lower inference cost than a frontier model alone.
Cost, Latency, and Maintenance Compared
Dimension | RAG | Fine-Tuning (LoRA) | Hybrid |
Initial setup time | 1 to 3 weeks | 4 to 8 weeks | 6 to 12 weeks |
Knowledge updates | Minutes (replace documents) | Days to weeks (retrain) | Minutes for RAG layer |
Per-query cost | Embedding + LLM API | LLM API only | Embedding + fine-tuned LLM |
Latency | Higher (retrieval adds 50 to 200ms) | Lower (no retrieval step) | Moderate |
Source citation | Yes (native) | No (not possible) | Yes (via RAG) |
Style consistency | Depends on prompt | High (baked in) | High (via fine-tuning) |
Model swappability | High (RAG is model-agnostic) | Low (adapter tied to base) | Moderate (RAG swappable, adapter needs retraining) |
Maintenance burden | Vector DB plus re-indexing | Training pipeline plus data curation | Both |
Eval complexity | Retrieval metrics plus generation metrics | Task-specific metrics | Both |
The key insight: RAG is cheaper to build and maintain, but caps at the quality of your retrieval. Fine-tuning is more expensive but locks in behavior. The hybrid gives you both at the cost of maintaining two systems.
Common Mistakes and How to Avoid Them
Mistake 1: Fine-Tuning to Teach Facts
This is the most common and most expensive mistake. You fine-tune a model on your product documentation, and it works for a month. Then the documentation updates. The model now answers with stale information, and you have no way to fix it without retraining. Use RAG instead.
Mistake 2: Skipping the Re-Ranker
Plain vector similarity (cosine top-k) is the largest preventable quality cap in production RAG. A cross-encoder re-ranker over the top 20 to 50 candidates improves answer quality by 15 to 30% in most benchmarks. Use Cohere Rerank, BGE Reranker, or Voyage.
Mistake 3: Naive Fixed-Size Chunking
Splitting documents every 500 tokens regardless of structure shreds context. A chunk that cuts off mid-sentence loses meaning. Use semantic chunking that respects document structure: paragraphs, sections, or natural boundaries. Add 15% overlap between chunks to preserve context across boundaries.
Mistake 4: No Evaluation Harness
Without a labeled test set and automatic metrics, you cannot tell whether a change helped or regressed. Use Ragas, TruLens, or DeepEval. Measure retrieval quality (recall, precision) separately from generation quality (faithfulness, relevance). Set up evaluation in week one, not after launch.
Mistake 5: Full Fine-Tune When LoRA Would Do
Full fine-tunes are slower, more expensive, and harder to roll back than LoRA adapters. Start with LoRA. Move to full fine-tuning only if LoRA cannot reach the required quality after extensive hyperparameter tuning. In 2026, LoRA handles 90% of fine-tuning use cases.
Mistake 6: No Hybrid Retrieval
Vector-only search misses exact keyword matches. A product code, a person name, or a specific error message may not have a close vector neighbor but is an exact keyword match. Use hybrid retrieval (BM25 plus vector) to catch both semantic and lexical matches.
Building the Evaluation Harness Before You Build the System
The single habit that separates teams who improve a RAG system from teams who argue about it is building the evaluation harness before the system. It sounds like a delay. It is the opposite: it is the thing that makes every later decision cheap.
Two harnesses, not one
A RAG system has two failure surfaces, and they need separate measurements.
Retrieval quality: did the correct passage reach the context window? Measure this with recall at K, precision at K and mean reciprocal rank against a set of labelled queries, exactly as you would for any search system.
Generation quality: given the correct passages, did the model produce a faithful answer? Measure this with a rubric that scores the answer against the retrieved passages, not against the world.
Keeping them separate is what lets you diagnose. A wrong answer that came from a missing passage is a retrieval bug. A wrong answer that came from a present passage is a generation bug. Collapsing the two into a single "answer quality" score leaves you unable to tell which half of the system to change.
Getting the queries
Take them from real traffic. A support inbox, a search log, a chat channel where people ask each other questions. Rewrite them minimally for privacy and leave the phrasing alone.
Then deliberately add the four query shapes that break systems:
a query in the user's vocabulary where the document uses internal jargon
a query whose answer spans two documents
a query about something that has been removed, where the correct answer is to say so
a query in a second language, if your corpus has any
The second and third categories are the ones that find real defects. A set made only of well-formed questions with single-document answers will report 0.95 recall and teach you nothing.
Metrics that behave
Retrieval. Recall at 10 decides answer quality, because a missing passage cannot be recovered downstream. Precision at 10 controls how much irrelevant context you pay for. MRR tells you whether the right answer is near the top, which matters when you only pass five passages to the model.
Faithfulness. The proportion of claims in the answer that are supported by the retrieved passages. This is the metric that catches hallucination, and it is measurable: split the answer into claims, check each against the context, report the ratio.
Answer relevance. Whether the answer addresses the question that was asked, as opposed to a related one.
Citation accuracy. Whether the citations the answer carries actually support the claims they are attached to. A system that cites the wrong passage is worse than one that cites nothing, because it transfers false confidence to the reader.
The threshold file
Write down the minimum acceptable score for each metric. A worked example for an internal documentation assistant:
recall_at_10 >= 0.90 precision_at_10 >= 0.40 mrr_at_10 >= 0.75 faithfulness >= 0.95 citation_accuracy >= 0.90 p95_query_latency_ms <= 2000
Six thresholds, and the harness exits non-zero when any of them fails. Now the harness is a gate rather than a report. A change that improves one metric by breaking another cannot ship quietly, because the failing threshold blocks it.
Where teams get this wrong
Scoring generation without checking retrieval first. If retrieval recall is 0.6, no amount of prompt engineering on the generation step will fix the answers. Fix the upstream layer first, then measure the downstream one.
A set that is frozen forever. The set should grow. Every production failure that a user reports should become a new case, added in the same change that fixes it. A set that never grows stops describing your system.
Using the model to grade itself. A model scoring its own output correlates weakly with human judgement. Use a different model as a judge, give it a written rubric, and spot-check a sample against a human label every few weeks.
Fine-Tuning in Practice: Data, Configuration and What to Measure
Fine-tuning is described in the abstract far more often than it is described concretely. This section is the concrete version: what data you need, what settings move the result, and what you measure afterwards.
How Much Data, and What Kind
The honest answer is that it depends on the task, and the useful rule is to measure rather than to ask. A working starting point for a behaviour task such as enforcing an output format or a tone:
200 to 500 examples enough to see whether the direction is right 500 to 2,000 examples a typical production adapter 2,000 to 10,000 diminishing returns unless the task is genuinely hard
A quality point that matters more than the count: examples must be varied. Two thousand examples generated from one template teach the model the template. Two hundred examples written by a person who solved the problem in different ways teach the behaviour.
Each example is an instruction and a response. Three properties decide whether the adapter works:
The responses must be exactly the output you want at inference. If the desired output is JSON, every training response is JSON with no prose around it.
The instructions must look like your real inputs, including the awkward ones. Training on clean questions and serving messy ones is a common and avoidable failure.
Include the negative space. If the model must decline some requests, the training set includes examples of declining.
What Actually Moves the Result
Learning rate. The highest-leverage parameter and the most common cause of a broken adapter. Too high and the model forgets its general ability while learning your format; too low and nothing changes.
Number of epochs. For a behaviour adapter, one to three passes over the data is normal. More passes over a small set teaches memorisation of the examples, which shows up as an adapter that reproduces training outputs and fails on new inputs.
Rank and target modules. The rank controls how much capacity the adapter has. Higher rank is not automatically better: it is better only when the task needs it, and it is worse when the data is small, because there is more room to overfit.
Sequence length. Long training examples are truncated at the configured length, silently. If your real inputs are long, verify that the pipeline is not cutting off the part that matters.
Overfitting an Adapter, and How to See It Coming
The signal: training loss keeps falling while held-out loss starts rising. Watch both curves, not just the training one. When the held-out curve turns upward, that is the point to stop.
The symptom in production: the adapter produces the training format correctly on inputs that resemble the training set, and something worse than the base model on inputs that do not. Because the training inputs resemble your real inputs, the failure appears only on the tail, which is exactly where you also have little test data. The defence is a held-out evaluation of at least 100 examples never used in training, plus a set of deliberately out-of-distribution inputs.
What to Measure After Training
Four measurements, and all four are necessary:
Measurement | What it tells you |
Task accuracy on held-out data | Did the adapter learn the behaviour |
Base-model accuracy on the same data | Is the adapter actually better than the prompt |
General ability on an unrelated task | Did the adapter damage what the model already knew |
Format compliance rate | Does the output always parse |
The second row is the one teams skip and the one that matters most. If a well-written prompt reaches 0.90 and the adapter reaches 0.91, the adapter has not earned its maintenance cost. The strongest case for an adapter is a large gap: a prompt at 0.72 on a format-heavy task where the adapter reaches 0.97.
Synthetic data, used carefully
Two rules keep generated training data honest: ground the generation in real inputs rather than invented ones, and never train on model output that a human has not sampled. Take a random sample, review it by hand, record the error rate. If the sample is 3 percent wrong, the adapter is being trained to be 3 percent wrong. Synthetic data is safe for behaviour and format, because format is verifiable. It is risky for facts, because a plausible wrong fact is indistinguishable from a right one without checking.
The baseline you must run
Before training anything, run the base model with three escalating prompts: a plain instruction, an instruction with two or three examples, and an instruction with a carefully written schema. Record each score. Many fine-tuning projects end at this step, correctly, because prompt engineering reached the target and an adapter was never needed. That is not a wasted week: it is a week that saved a training pipeline, a data curation process and an ongoing maintenance burden.
The Cost Model: Doing the Arithmetic Before You Build
Both approaches have a cost, and the cost is computable before you write anything.
RAG, per year embedding the corpus corpus tokens x price, once plus on every re-index query embedding queries per day x query tokens x price retrieval compute queries per day x index size factor generation queries per day x (context + output tokens) x model price Fine-tuning, per year dataset creation hours of human time x rate, plus review training run GPU hours x instance price, per attempt, not per success evaluation hours x rate, per iteration hosting a GPU or a hosted endpoint running continuously re-training every data change, every base model upgrade
Two structural facts follow. RAG's cost is mostly the language model you were going to call anyway, plus a modest retrieval overhead, because the generation line dominates. Fine-tuning's largest recurring line is usually hosting rather than training, because a fine-tuned model is a model you run. If you fine-tune because the API was too expensive at volume, the arithmetic works only if your volume amortises a dedicated endpoint.
Why retrieval wins on facts
Ask one question about the thing you want the model to have: will it still be true in a year?
If no, retrieve it. Store one new document and it becomes available to every future query at an embedding call that costs a fraction of a cent. Teach one new fact by fine-tuning and the marginal cost is a training run, an evaluation cycle, a deployment and a regression risk on everything the adapter already learned. Retrieval has a per-fact cost near zero; fine-tuning has a per-training-run cost that does not shrink as the corpus grows.
Retrieval one hundred document updates one hundred embedding calls, minutes stale answers after a change zero, once the index is updated Fine-tuning one hundred document updates the same, a hundred times stale answers after a change every answer until the next training run
Fine-tuning wins economically in exactly one place: a stable behaviour at high volume. If a narrow transformation happens ten million times a month and a small tuned model can produce it, the per-call saving recurs and the maintenance cost is bounded. Knowledge is not stable, and volume does not reduce the cost of a retraining cycle.
The line teams forget
Maintenance. A fine-tuned adapter is tied to a base model, so a base-model upgrade means retraining, re-evaluation and re-deployment. A RAG pipeline is model-agnostic: swap the language model and the whole retrieval layer is unchanged. Over a two-year horizon that difference is often larger than the difference in training cost. Fine-tune for a behaviour you expect to be stable for a long time; retrieve for knowledge that will change.
Failure Triage: Which Layer Is Actually Broken
A bad answer has a cause, and the cause determines the fix. Working through this in order saves the most wasted effort of any habit in this lecture.
Step 1: Was the right passage retrieved at all?
Look at the candidate set the first stage returned, before reranking. Three outcomes:
The passage is absent. The failure is retrieval. Do not change the language model or the prompt's instructions. The candidates are the embedding model, the chunking, the query formulation, or a metadata filter that excluded the right document.
The passage is present but ranked low. The failure is ranking. The reranker, or the absence of one, is the place to work.
The passage is present and top-ranked. The failure is generation. Now the prompt and the model are the right things to examine.
Step 2: If retrieval failed, narrow it further
Symptom | Likely cause | First fix |
The document is not in the index | Ingestion gap, or a filter excluded it | Check the ingestion log and the filter |
Indexed but never retrieved | Vocabulary mismatch between query and chunk | Prepend the section heading to the chunk |
Retrieved for some queries only | Chunking split the answer across two chunks | Structure-aware chunking, or more overlap |
Retrieved, then filtered out | Filter applied after the search | Iterative scanning, or pre-filter |
Retrieved for the wrong reason | The chunk is on the topic but does not answer it | Split the chunk, or add a reranker |
Step 3: If generation failed, separate the two failures
Unfaithful. The answer contradicts or goes beyond the retrieved passages. This is a grounding failure. The fixes are prompt-level (answer only from the context, and say when the context is insufficient) and architectural (fewer passages, and citations required).
Faithful but wrong. The answer correctly reflects passages that are themselves wrong or outdated. This is a data problem, not a model problem. Fix the corpus: remove the stale document, or add the date to the chunk so the model can prefer the current one.
Step 4: When the failure is behavioural rather than informational
Sometimes the passages are correct, the answer is accurate, and it is still wrong: the format is wrong, the tone is off, the output does not parse. This is the one class of failure where fine-tuning is the right answer. The model has the information and the capability; what it lacks is your convention. A prompt can usually express the convention, which is why the prompt comes first. When a prompt cannot hold the convention reliably across the whole input distribution, the convention is a behaviour, and behaviour is what an adapter is for.
A Decision Procedure You Can Run in an Afternoon
The comparison tables in this lecture describe the shape of the problem. This section is the procedure to actually run, in order, on a real use case.
Step 1: Write the task as one sentence with a verb
"The system must extract the termination clause from a contract and return it as structured JSON."
If the sentence contains two verbs, it is two tasks and should be split before anything else is decided.
Step 2: Classify the requirement
Ask which of the following the system actually needs:
Information the model was not trained on (your documents, your data, today's values). This is retrieval.
A transformation the model can perform but not consistently (a strict output format, a register, a long fixed structure). Reach for the prompt first, then consider an adapter.
Reasoning the base model does poorly (a specialised extraction or classification problem). Consider an adapter after the prompt baseline.
A model that is too large or too expensive for the volume (a narrow high-volume task). Consider distillation.
Most real systems need the first, and only the first. Writing down which of the four you are solving prevents the most common error, which is solving the second because the first is not yet built.
Step 3: Set the baseline before the architecture
Measure the simplest possible system on your evaluation set. That is the language model, your prompt, and no retrieval at all. Record the score.
This baseline is not a formality. It tells you how much room there is. If the no-retrieval baseline already scores 0.85 on your set, the retrieval layer is adding something modest and should be sized accordingly. If it scores 0.30, the retrieval layer is the system.
Step 5: Only now consider behaviour
With retrieval working, look at the residual failures. Sort them:
failures where the correct passage was missing: retrieval, still
failures where the passage was present and the answer was unfaithful: grounding, a prompt problem
failures where the answer was correct and the format was wrong: a behaviour problem
Only the third class justifies an adapter. If the third class is empty, you are done, and the correct engineering decision was to build nothing else.
Reader exercise
Take one task you have built or are planning to build. Write the six steps above as six lines, and fill in the actual number for step 3 if you can. If you cannot, that missing number is the next thing to produce, and it is worth more than any architecture decision you could make today.
Feynman Summary: Explain It Like You Are 12
Imagine you have a smart friend who has read every book in the library. That friend is the language model. Now you want your friend to answer questions about your school textbook.
RAG is like giving your friend the textbook and saying "look up the answer in here before you respond." Every time you ask a question, your friend opens the book, finds the right page, reads it, and gives you the answer. If the textbook gets updated, your friend automatically uses the new version because they check the book each time.
Fine-tuning is like sending your friend to a training camp where they learn to answer in a specific way: always use bullet points, always sound professional, always format the answer as a table. The training camp does not teach new facts. It teaches a style of answering.
Hybrid is doing both: your friend goes to the training camp to learn the style, and still checks the textbook for the facts. That gives you the best answers: correct facts, delivered in the right format.
The big mistake is sending your friend to training camp to memorize the textbook. They might remember some of it, but the moment the textbook changes, their memory is wrong, and you have to send them back to camp. Just let them check the book each time instead.
Mindmap: The Complete Picture
This mindmap shows the full decision tree: start with the problem type (knowledge or behavior), follow the branch to the recommended approach (RAG, fine-tuning, or hybrid), and see the tools, costs, and trade-offs at each node.
UNOP Sound (University 365 Neuroscience Oriented Pedagogy)
Take five minutes to consolidate your memory. Play the isochronous tone track (10Hz alpha frequency) with your eyes closed. Alpha-frequency tones after a learning session support consolidation, helping move what you just learned from short-term to long-term memory.
[Audio player: UNOP Post-Lecture Isochrone (10Hz, 5 minutes)]
Practical Exercise: Build a Decision Matrix
Take a project you are working on (or imagine one) and walk through this decision matrix. Write your answers down.
Step 1: Define the Problem
Write one sentence describing what your LLM application needs to do. Example: "Answer customer support questions using our help center documentation."
Step 2: Answer These Questions
1. Does the knowledge change frequently (daily, weekly, monthly)? Yes or No.
2. Do you need to cite sources or show provenance? Yes or No.
3. Is your corpus larger than 200,000 tokens? Yes or No.
4. Does the model already produce correct facts but in the wrong format or tone? Yes or No.
5. Do you need a small, fast model for high-volume inference? Yes or No.
6. Do you have 500 or more curated input-output examples for fine-tuning? Yes or No.
Step 3: Apply the Decision Rules
- If you answered Yes to questions 1, 2, or 3: start with RAG.
- If you answered Yes to questions 4 or 5: consider fine-tuning.
- If you answered Yes to both groups: build the hybrid (RAG first, then fine-tune the residuals).
- If you answered No to everything: start with prompt engineering and revisit only when you hit a wall.
Step 4: Estimate Costs
- RAG build: 1 to 3 weeks of engineering time, plus embedding and vector database costs.
- Fine-tuning build: 4 to 8 weeks, including data preparation, GPU training, and evaluation.
- Hybrid: 6 to 12 weeks for the full system.
Write down your estimated timeline and budget. This exercise gives you a concrete starting point for your next LLM project discussion.
Glossary
Term | Definition |
RAG (Retrieval Augmented Generation) | Technique that retrieves relevant documents from an external index and passes them to the LLM as context at query time, without modifying model weights. |
Fine-Tuning | Training process that adjusts model weights on a dataset of input-output examples to change the model behavior, style, or format. |
LoRA (Low-Rank Adaptation) | Fine-tuning method that trains a small adapter module (under 1% of parameters) on top of a frozen base model. Cheap, fast, and reversible. |
QLoRA | Extension of LoRA that quantizes the base model to 4-bit, reducing memory requirements enough to fine-tune large models on a single GPU. |
DPO (Direct Preference Optimization) | Training method that fine-tunes models on preference pairs (output A is better than output B) without a separate reward model. Simpler than RLHF. |
RLHF (Reinforcement Learning from Human Feedback) | Training method using a reward model and reinforcement learning to align model outputs with human preferences. Largely replaced by DPO in 2026. |
Embedding | Vector representation of text that captures semantic meaning. Used to find similar passages by computing vector similarity. |
Vector Database | Specialized database that stores and searches high-dimensional vectors. Examples: Pinecone, Weaviate, Qdrant, pgvector. |
BM25 | Sparse keyword matching algorithm that scores documents by term frequency and inverse document frequency. Used in hybrid retrieval alongside vector search. |
Re-Ranker | Cross-encoder model that re-orders retrieved candidates by true relevance to the query. The biggest single quality lever in RAG pipelines. |
Chunking | Process of splitting documents into smaller passages for embedding and retrieval. Semantic chunking respects document structure; naive chunking uses fixed token counts. |
Hybrid Retrieval | Search approach combining dense vector similarity and sparse keyword matching (BM25) to catch both semantic and lexical matches. |
Distillation | Fine-tuning a small model on outputs from a larger frontier model to achieve similar quality at lower cost. The strongest commercial case for fine-tuning in 2026. |
Context Window | Maximum number of tokens a model can process in a single prompt. Frontier models in 2026 support up to 1M tokens. |
Prompt Caching | API feature that caches static prompt prefixes at reduced cost (approximately 10% of normal input cost), making long-context strategies economically viable. |
Agentic RAG | RAG pattern where the model decides when and what to retrieve, decomposes complex queries into sub-queries, and self-corrects when retrieved evidence is weak. |
GraphRAG | RAG variant that combines knowledge graphs with vector search for entity-heavy, multi-hop reasoning queries. |
Evaluation Harness | Automated testing framework that measures retrieval and generation quality on a labeled dataset. Examples: Ragas, TruLens, DeepEval. |
Hallucination | Model output that is fluent and confident but factually incorrect. RAG reduces hallucinations by grounding answers in retrieved context. |
UNOP | University 365 Neuroscience-Oriented Pedagogy: the teaching framework behind this lecture format, using brain-state preparation, microlearning, and spaced consolidation. |
Quiz: TEST YOUR UNDERSTANDING
1. Your company's product catalog changes weekly. You need a chatbot that answers questions about current products and prices. Which approach should you use first?
A) Fine-tune a model on the product catalog
B) Use RAG with the product catalog as the knowledge base
C) Use prompt engineering with the full catalog in the system prompt
D) Wait for the catalog to stabilize before building anything
2. What is the primary difference between RAG and fine-tuning?
A) RAG is faster to build, fine-tuning is more accurate
B) RAG changes what the model knows, fine-tuning changes how the model behaves
C) RAG uses GPUs, fine-tuning does not
D) RAG works with open-source models, fine-tuning works only with proprietary models
3. Which scenario is the strongest commercial case for fine-tuning in 2026?
A) Teaching a model your company's internal policies
B) Distilling frontier model performance into a smaller, cheaper model for a narrow task
C) Making the model aware of real-time stock prices
D) Building a customer support chatbot that cites sources
4. What is the most common mistake in RAG pipelines?
A) Using too many embeddings
B) Skipping the re-ranker step
C) Using a vector database instead of a relational database
D) Fine-tuning the embedding model
5. In the hybrid RAG plus fine-tuning pattern, what should you do first?
A) Fine-tune the model, then add RAG
B) Build RAG, identify behavioral failures, then fine-tune for those residuals
C) Fine-tune and build RAG simultaneously
D) Neither: use prompt engineering only
Answers: 1-B, 2-B, 3-B, 4-B, 5-B
Related Resources
U365 INSIDE Publications
- How LLMs Actually Work: Transformers in 20 Minutes (AI Foundations, Lecture 1)
- Vector Databases Explained: Embeddings for Search (AI Engineering, Lecture 4, coming soon)
- Prompt Engineering at Production Scale (AI Skills, Lecture 5, coming soon)
External Resources
- Attention Is All You Need (Vaswani et al., 2017): the original transformer paper
- LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)
- Direct Preference Optimization (Rafailov et al., 2023)
- Ragas: evaluation framework for RAG pipelines
- Cohere Rerank documentation
- Qdrant, Weaviate, Pinecone, pgvector: vector database options
Related U365 Lectures (Coming Soon)
- Building Your First AI Agent with Function Calling (AI Agents, Lecture 3)
- Model Quantization: Running LLMs on Your Laptop (AI Engineering, Lecture 7)
- The AI Stack 2026: What Every Developer Needs (AI Engineering, Lecture 10)
U.Copilot for This Lecture
Copy and paste this prompt into the U.Copilot AI Agent on university-365.com to explore this topic further:
I just completed the U365 INSIDE Lecture "RAG vs Fine-Tuning: When to Use Each" from UIT. I want to apply this to my own project. Help me: 1. Describe my use case in one sentence 2. Walk me through the decision matrix from the lecture 3. Recommend whether I should use RAG, fine-tuning, or a hybrid approach 4. Suggest specific tools and libraries for my recommended approach 5. Estimate the timeline and resources I will need My use case is: [describe your project here]
Next Steps
1. Take the quiz above and check your answers at the bottom of this section.
2. Complete the Practical Exercise: build a decision matrix for a real or imagined project.
3. Read the next lecture in the AI Engineering series: Vector Databases Explained.
4. If you have not completed Lecture 1 (How LLMs Actually Work: Transformers in 20 Minutes), start there for the foundational architecture.
5. Visit university-365.com/uit to explore UIT programs in AI Engineering and Data Science.
6. Try the U.Copilot prompt above to get personalized recommendations for your project.
IMPORTANT NOTICE
Copyright University 365, Inc. All rights reserved.
This lecture is part of the U365 INSIDE Lectures series, produced by UIT (University 365 Institute of Technology) under the UDA Department of Academics. The content follows the UNOP (University 365 Neuroscience-Oriented Pedagogy) framework and the 5M2S (5 Minutes to Success) microlearning format.
All lectures in this series are free to access. For enrollment in UIT degree programs, certificate programs, or executive education, visit university-365.com/tuition.
For permissions or inquiries, contact uda@university-365.com.
This content is for educational purposes. Technical details about specific tools, pricing, and APIs reflect publicly available information as of September 2026 and may change. Always consult official documentation before making architecture decisions for production systems.
Published by the Department of Academics, University 365.
Lecture delivered by the University 365 Institute of Technology (UIT).
Sam Utteker, Dean of Technology, UIT
Signed for the academic year 2026.
Comments