Fine-Tuning or Context Engineering? Stop Flipping the Coin

Most teams pick between fine-tuning and context engineering the way they pick a lunch spot. Someone has a preference, nobody wants to argue, and the loudest voice wins. Then six weeks later the bill arrives, the model is hallucinating customer data, and everyone is surprised.

The choice deserves better than a coin flip, because the two approaches fail in completely different ways and the research on when each one works is now solid enough to reason from. This post stakes a position. Most of the time you should reach for context engineering first, fine-tune second, and only for the specific jobs fine-tuning is good at. Here is the evidence for why, and the cases where the opposite is true.

Start by naming the approaches precisely, because half the disagreements in this debate come from people comparing different things.

The broader discipline is about managing the whole context window as a finite resource, which is a useful way to think about it that we will come back to. The reason this distinction matters: fine-tuning is the only one of the two that touches the model itself. Everything else is about feeding a fixed model better inputs. That single fact drives most of the tradeoffs below.

Here is the finding that should reshape how you think about this, and the one most teams have never read.

A 2024 study from researchers at Google and the Technion, presented at EMNLP, asked a direct question: does fine-tuning a model on new factual knowledge make it hallucinate more? They built a controlled setup around closed-book question answering and varied how much of the fine-tuning data introduced facts the model did not already know. Two results came out of it.

Read that again, because it cuts against the most common reason teams reach for fine-tuning. People fine-tune to "teach the model our data." The research says that is the one thing fine-tuning is worst at. When you cram unfamiliar facts into the weights, you are not building a more knowledgeable model. You are training it to confidently state things, including things that are wrong, because you rewarded that pattern during training.

Now the other side. If facts do not belong in the weights, where do they go? Into the context, at inference time, where the model can read them rather than recall them.

The most cited practical comparison is a 2024 study from Microsoft researchers that ran fine-tuning and RAG head to head on an agricultural knowledge task, across Llama 2 13B, GPT-3.5, and GPT-4. The two techniques were cumulative, and the combination beat either one alone.

But the framing of "both" hides an order of operations, and the order matters. RAG and context engineering give you something fine-tuning structurally cannot: the ability to update knowledge by updating an index instead of retraining a model. Change a document, re-embed it, and the system is current on the next query. No training run, no eval cycle, no ML engineer in the loop.

“

For any knowledge that changes - which in most businesses is most knowledge - that operational difference is decisive.

The naive version of context engineering is "models have million-token windows now, so just put everything in." This is wrong, and there is now hard data on exactly how wrong.

A 2025 study from Chroma tested 18 frontier models, including GPT-4.1, Claude, and Gemini, on how performance changes as input length grows. Every single model degraded as the context got longer, and not gracefully. The decline showed up well before the models hit their advertised token limits.

The term that has stuck is context rot, and the key insight is that capacity is the wrong thing to optimize. What matters is the signal-to-noise ratio of what you put in front of the model, not how much you can technically fit. This builds on an older and equally important result. The "Lost in the Middle" work showed that models reliably miss information buried in the middle of a long input, favoring content at the very start and the very end - a U-shaped accuracy curve.

Stack context rot on top of "lost in the middle" and the lesson is clear.

“

A context window is not a bucket you fill. It is an attention budget you spend - and spending it badly makes the model dumber even when every fact it needs is technically present.

This is the whole reason context engineering is a discipline and not just "paste the docs in." Good context engineering means retrieving precisely, ranking what matters to the top, discarding exploration traces and stale turns, compressing aggressively, and keeping the window lean. The teams that treat the context window as a scarce resource get reliable systems. The teams that treat it as free storage get systems that pass the demo and fall over in week two.

Everything above is a case for reaching for context first. But fine-tuning is not a trap to avoid, it is a tool with a narrow and real purpose. The research points to where it earns its keep.

Here is the position, made concrete. Walk these in order.

Fine-tune when you have a specific behavioral reason and the evidence to back it, not because it feels like the more serious engineering choice.

The reason this debate stays unsettled is that both camps are usually right about their own use case and wrong to generalize it. The fine-tuning advocates built something where behavior mattered and fine-tuning delivered. The RAG advocates built something where knowledge changed and retrieval delivered. Then each assumed their answer was the answer.

“

Stop flipping the coin. The decision is not about taste - it is about whether your problem is a knowledge problem or a behavior problem, and the literature is clear enough now that you can usually tell which one you have before you write a line of code.