Large reasoning models (LRMs) have gained reasoning power by flexibly switching between different reasoning trajectories. But this flexibility has a downside: models often switch too frequently, which can lead to under-thinking—shallow or incomplete reasoning.
The paper proposes SteadyThought, a thought-level preference optimization approach. Instead of optimizing only final answers, it trains models to prefer better individual thoughts, encouraging more sustained and coherent reasoning paths.
By directly shaping the thought process, SteadyThought aims to reduce excessive switching and promote steadier reasoning. The authors position this as a complement to existing methods that focus on outcome-level optimization.