Suppose you are building a documentation assistant. Users ask questions about your product’s API, and the assistant must answer from your docs. You have two main ways to give the model the material it needs.
The first is retrieval-augmented generation, usually shortened to RAG. Your code splits the documents into smaller pieces called chunks, searches for the chunks most related to the user’s question, and sends only those chunks to the model along with the question. The second is a long-context prompt. Your code sends the whole body of relevant material in each request and lets the model read all of it.
Both approaches are limited by the model’s context window. This is the maximum amount of text the model accepts in one request. It is measured in tokens, the units of text that the model reads and that providers bill for. In the 2024 comparisons below, long prompts often scored higher when the full input fit. Retrieval becomes useful when the corpus cannot fit in one request or when your code needs to select a small, permission-checked part of it. Freshness and cost depend on how you update and assemble either system. Retrieval also has its own failure modes. For that reason, test the choice against your own questions before you commit.
How the two approaches differ
A RAG pipeline has two stages. First, an indexing step splits documents into chunks and stores a numeric representation of each chunk, called an embedding, in a search index. Second, at question time, the retriever compares the question against the index and returns the highest-scoring chunks. The model then answers from those chunks alone. The retriever decides what the model sees, so the model never reads the passages the retriever skipped.
A long-context prompt has one stage. Your code assembles the material, adds the question, and sends it. The model can read everything in the prompt, but it must find the relevant part on its own. As the next sections show, both the retriever’s choices and the model’s attention to a long prompt can fail.
What the benchmarks show
The most detailed head-to-head comparison comes from a 2024 study by Zhuowan Li and colleagues, published in the EMNLP 2024 Industry Track. The authors tested Gemini-1.5-Pro, GPT-4o (version gpt-4o-2024-05-13), and GPT-3.5-Turbo (version gpt-3.5-turbo-0125) on English question-answering datasets. Their default RAG setup split documents into 300-word chunks and passed the five highest-scoring chunks to the model. When the model had enough room for the full input, long context scored higher on average for all three models. The average gap ran from 3.6% for GPT-3.5-Turbo to 13.1% for GPT-4o.
The same study also found a case where RAG was the only option. GPT-3.5-Turbo has a 16k-token window, and two of the test datasets averaged about 147k words. On those datasets, RAG beat long context, because the full input did not fit. The authors describe this as a specific use case of RAG.
A second study by Li, Cao and colleagues, posted as an arXiv preprint, reached a similar conclusion across 12 datasets. Long context answered 56.3% of questions correctly, and RAG answered 49.0% correctly. The two methods also failed on different questions. Long context alone answered 2,287 questions correctly that RAG missed, and RAG alone answered 1,294 that long context missed. That overlap means a retrieval step can rescue some questions that a long prompt gets wrong, and the reverse is also true.
The preprint also found that question type changes the winner. Long context did better on "who," "where," and "which" questions, and RAG did better on yes/no questions. The retriever mattered too. In that study’s sample set, the best retriever (RAPTOR) returned the correct answer 38.5% of the time, while the weakest (Contriever) managed 20.1%. A RAG system is only as good as its retriever, and the retriever is a choice you make.
These results do not settle the question for every product. The LaRA benchmark, which tested 2,326 cases, concluded that neither approach is a "silver bullet." It found that the better choice depends on model size, context length, task type, and the characteristics of the retrieved chunks. Earlier studies also disagree with each other. These studies tested different model generations, including models from 2023 and 2024. Check whether their results hold for the models you use.
The benchmarks also show that test design can flip the result. The EMNLP authors ran a passkey retrieval test, in which the model has to find a short code hidden in long text. On Gemini-1.5-Pro, RAG scored 80.34% on the original wording. When the question asked about a "special token" instead of a passkey, RAG fell to 4.58%, while long context stayed at 69.32%. A single rephrased question changed the apparent winner. Your evaluation needs questions that look like the questions your users actually ask.
Freshness and permissions
Freshness is the first place where retrieval has a structural advantage. A perspective paper by Wang and colleagues argues that RAG addresses models trained on static data that struggle with rapidly changing information or domain-specific queries. The same paper lists personalized and private information access as scenarios that call for RAG.
In practice, the two approaches handle freshness differently. A long prompt that your code builds from current files on every request always reflects the latest text in those files, because each request reads them fresh. A RAG index is a snapshot. When an API page changes, the chunks for that page must be re-embedded and written to the index, or the assistant will answer from the old text. The research does not measure how long that update takes in production, so the update process needs its own test: change a page, then confirm that a question about the change retrieves the new text.
Permissions work differently. A long prompt contains whatever your code loads, so a user who lacks access to an internal page must never have that page loaded into the prompt. Retrieval gives you a natural place to enforce access. You can apply the user’s permission filter inside the search query, so chunks the user cannot see never reach the model. Putting "do not reveal internal pages" in the prompt is not an access control, because the model can still read whatever it receives. I recommend enforcing permissions in the retrieval filter or in the code that builds the prompt. Then the model reads only text that has already passed the check.
Four ways retrieval fails
Retrieval fails in four common ways. Each one affects a documentation assistant differently.
The first is a missed chunk. The retriever scores each chunk by how similar it looks to the question. The chunk that answers the question can score lower than a chunk about a related topic. The EMNLP study found that raising the number of retrieved chunks helps but shows diminishing returns. For Gemini-1.5-Pro, RAG scored 20.24 with one chunk and 37.92 with five, and at 50 or more chunks all methods converged to similar results. Sending more chunks does not reliably close the gap, so you should measure how often the needed passage appears in the top results.
The second is a multi-step question. A question such as "which timeout applies to the webhook that the OAuth flow calls" may need one chunk to name the webhook and a second chunk to give its timeout. A single retrieval pass can find the first chunk and miss the second. The EMNLP authors traced most RAG failures on multi-hop Wikipedia datasets to this multi-step retrieval problem.
The third is a question that needs the whole document. Some questions ask for an overview, a comparison across sections, or an implied conclusion. The EMNLP authors found that RAG failed most often on implicit questions, which need an understanding of the whole document. In their tests, these were NarrativeQA questions.
The fourth is a broken chunk. A fixed-size split can cut a procedure in half, so the first chunk holds steps one through three and the next holds steps four and five. Li and colleagues warn that chunking can break continuity and hurt RAG results. Splitting on section headings keeps each procedure intact, and some teams add a small overlap between neighboring chunks so a sentence cut at a boundary still appears whole in one chunk. The research tested fixed-size chunks, so the heading-based split is my recommendation for documentation rather than a finding from these studies. Checking a few chunks by eye before you ship the index catches most boundary problems quickly.
A long prompt has its own version of this problem. The Lost in the Middle paper by Liu and colleagues found that models answer most accurately when the relevant information sits near the beginning or end of the input. Accuracy drops when the information sits in the middle, and the authors reported this even for models built for long inputs. The study used 2023-era models, but the pattern is a reason to check where the answer lands in a long prompt. If you paste a long reference set, put the most likely relevant section first, or split the material and answer in stages.
When a small complete document is the better choice
A short, complete document often beats retrieval. If the answer to a common question lives on one page, such as an authentication guide, sending that page whole gives the model the procedure in its original order and removes the retriever as a failure point. Retrieval adds a search step, an index to maintain, and a chance to miss the one chunk that matters, and none of those steps helps when the whole page fits.
A reasonable rule for a documentation assistant is to load the full text when the questions in an area can be answered from a small set of pages and the combined text fits comfortably in the model’s window. Use retrieval when the full corpus is too large, when it spans many products or permission levels, or when it changes often enough that a snapshot index would go stale without a managed update process. You can also load a full page for a question that the retriever has already matched to that page, which blends the two approaches.
Cost and latency
Cost is the most measurable difference. The EMNLP authors note that RAG significantly reduces input length, and that API pricing for these models is based on input tokens. Their comparison puts RAG at roughly 17% of long context’s input-token volume, which the paper uses as a cost proxy. The authors also state that long-context prompting stays expensive because the compute needed for transformer models grows quadratically with the number of input tokens, which means doubling the input more than doubles the work.
Prompt caching changes the math for repeated material. The OpenAI prompt caching guide explains that cached input tokens are billed at a lower rate than uncached ones. The guide’s worked example assumes $1 per million input tokens and $0.10 per million cached tokens. These are illustrative rates. Use your model’s current rates when you calculate costs. If your assistant sends the same documentation prefix with every question, caching can narrow the gap between the two approaches. Caching works only when the start of the prompt matches exactly. If the start changes on each request, the cache never hits.
A hybrid method can reduce cost without a large loss in quality. In the EMNLP paper, the SELF-ROUTE method sends the top retrieved chunks to the model first, and lets the model reply "unanswerable" when the chunks are not enough. Only those unanswerable questions go to the full long-context prompt. The retrieval step used about 1.5k tokens, compared with 10k to 100k tokens for the full contexts, and about 82% of Gemini-1.5-Pro’s questions were answered in the retrieval step. SELF-ROUTE cut cost by 65% for Gemini-1.5-Pro and 39% for GPT-4o. The paper describes the scores as close to long context, although its tables and prose give different average gaps. The trade-off is that a question that falls back to the long prompt costs two calls instead of one. The saving therefore depends on how often the first pass succeeds on your traffic.
The EMNLP cost analysis counts tokens and does not measure latency. A long prompt takes longer to process because the model reads more text, and a RAG system adds a search step before the model call. Neither the study nor the pricing pages give a latency figure that applies to your deployment, so you need to time both paths on your own infrastructure.
Keeping answers traceable
Citations matter for a documentation assistant, because a developer who gets an answer about a timeout setting needs to check the source. Neither the benchmarks nor the pricing pages address citation quality, so the guidance here is my recommendation.
With RAG, each chunk can carry metadata such as the page path and heading. Your code can pass that metadata to the model, so the answer can name the section it drew from, and your interface can link to that section. With a long prompt, the model must point to a location in a large text, and a reader has to search the full text to verify it. In both cases, check the cited passage. A model can name a source that does not support the claim it attached to that source.
Testing the choice on your own questions
The benchmarks show that the answer depends on the task, so run your own comparison before you commit to either approach. Collect real questions from users or support logs, and include questions from each failure category above: direct lookups, multi-step questions, yes/no questions, overview questions, and questions phrased differently from the page headings. The Li, Cao and colleagues preprint notes that many earlier studies used only about 200 questions per dataset, which makes results less reliable, so use a larger set where you can.
Score the two stages separately. For each question, record whether the needed passage appeared in the retrieved chunks, and whether the final answer was correct. A retrieval miss and a reading error need different fixes, and a combined score hides which one you have. Run both approaches on the same questions, and record the input tokens for each request so you can compare cost directly. Repeat the test when you change the model, because a model upgrade can shift the results.
Key Takeaways
- Check whether the full material fits the window first: If the material exceeds the model’s context window, retrieval or a hybrid is required, and the EMNLP results show RAG working where long context could not.
- Enforce permissions in the retrieval filter: Filter chunks by the user’s access rights before they reach the prompt, because an instruction to the model does not restrict what it reads.
- Split documentation on headings rather than fixed lengths: Keeping each procedure in one chunk avoids the broken-step failures that fixed-size splitting can cause.
- Measure retrieval and answers separately: Record whether the needed passage was retrieved, then whether the answer was correct, so you know which stage to fix.
- Price the repeated prefix before choosing: A cached long prefix can narrow the cost gap with RAG, so compare costs using current prices and your actual request pattern.
