Suppose a student knows three valid ways to solve a problem. They can use algebra, draw a diagram, or reason through the numbers directly.
Now imagine two ways of teaching the next set of problems.
In the first approach, the teacher repeatedly shows one worked solution and asks the student to reproduce that method. The student gets better at the task, but may increasingly default to the demonstrated method even when the other approaches they already knew would work.
In the second approach, the student attempts the problems using their existing methods. The teacher mainly provides feedback on whether the answer and reasoning are correct. If algebra works, it is reinforced. If the diagram works, that is also reinforced. Incorrect approaches are corrected.
The student is still learning, but the learning process does not require replacing several useful strategies with one preferred strategy. The learning is more generalized.
This is the type of distinction the RL’s Razor paper makes between SFT and on-policy RL.
a. Suppose a model already has three successful ways to solve a coding problem.
• Strategy A has 40% probability.
• Strategy B has 35%.
• Strategy C has 20%.
• Incorrect behavior has 5%.
With SFT, the training data may contain only Strategy B. The objective keeps increasing the probability of B, even though A and C were already valid solutions.
The model improves on the new task, but its behavior can change more than necessary.
b. With on-policy RL, the model generates trajectories from its own current behavior. If A, B, and C all succeed, all three can receive positive reward. The model can reduce the unsuccessful behavior while preserving much more of its existing distribution (probabilities) over successful strategies.
This does not mean RL is inherently better than supervised learning.
If we can construct an SFT datasets that preserved the model’s existing distribution over good behaviors while removing the bad ones, supervised learning could achieve the same objective.
The broader principle is to change the model only as much as necessary to learn the new capability.