Three Stages of Post-Training

Model capabilities are shaped in three stages-pipeline:

1. pre-training — “read ten thousand books”: next token prediction on internet texts, knowledge and language;
2. SFT — “teacher shows an exemplary solution”: thousands of input-output pairs, format and protocol, cheap and fast;
3. RL — “solve problems through trial and error” yourself: strategy and generalization, expensive.

SFT is mathematically the same token prediction task, only on different data and with loss only on the answer.

The order is not arbitrary: first form, then spirit.

Related: SFT memorizes - RL generalizes, First SFT then RL, Data and environment are more important than the algorithm