Prompt, retrieval or fine-tuning

When a finished language model does not do what you want, there are three options. They are not competitors. They are different tools for different problems.

The three, briefly

• Prompting: telling the model, right now, what to do.
• Retrieval (RAG): bringing in the facts it needs.
• Fine-tuning: changing how the model behaves by default.

Prompting

A prompt is the instruction and the context handed to the model: the user's question, the application's rules, examples, documents and descriptions of the available tools.

Always try the prompt first. It is the fastest to change, the cheapest to test and the easiest to undo. A surprisingly large share of problems that look like they need a different model turn out to be solved by writing the instruction more precisely and adding two examples of a good answer.

Retrieval

Retrieval means the system looks something up before answering (in the company's own documents, say) and hands the passages it found to the model.

Retrieval:

• does not usually change the model's weights
• lets the system use current or company-specific information
• can make it possible to show sources
• requires a good search and proper access control.

The point worth remembering: the quality of a retrieval system is usually decided by the search, not by the model. If the search returns the wrong passages, no model will rescue the answer. Access rights also have to be handled at the search stage, because the model cannot work out for itself who is allowed to see what.

Fine-tuning

In fine-tuning, the model's weights are adjusted for a particular purpose. It is how you teach a house style, a classification task or the conventions of an industry.

Fine-tuning is a poor way to add information. If the goal is for the model to know a price list that changed yesterday, retrieval is the right tool. Fine-tuning is the right tool when the model needs to behave consistently differently from the way it behaves by default.

The order that works

1. Write the prompt properly and try it on a smaller model.
2. If the answers are missing information, add retrieval.
3. If the style, format or task type will not stay in line even with instructions, consider fine-tuning.
4. Measure every step against the same test set, so you know which version is actually better.

Without the fourth step, the first three are guesswork.