Three Stages of Post-Training

Model capabilities are shaped in three stages – a pipeline: pre-training – “read ten thousand books” (predicting the next token on internet texts, knowledge and language); SFT – “teacher shows an exemplary solution” (thousands of input-output pairs, format and protocol, cheap and fast); RL – “solving problems through trial and error yourself” (strategy and generalization, expensive). SFT is mathematically the same token prediction task, only on different data and with loss only on the answer. The order is not arbitrary: first form, then spirit.

Related: [SFT memorizes - RL generalizes], [First SFT then RL], [Data and environment are more important than the algorithm]