SFT First, then RL

RL requires parsable output: if the model outputs chaotic text instead of JSON, the reward function cannot calculate anything – there is nothing to learn from. SFT plays the role of “first teach it to speak clearly”: a small number of demonstrations stabilizes the format. The reverse order does not work – the reward signal turns into noise. Boundaries of the rule: with a sufficiently strong base model, you can go straight to RL (DeepSeek-R1-Zero), but at the cost of readability – that’s why R1 still added a cold start SFT. Form first, then spirit.

Related: [Three stages of post-training], [SFT memorizes - RL generalizes]