A new arXiv preprint asks a pointed question: can a network keep learning while its training cross-entropy stays nearly flat at a positive level? The authors report that the answer is yes. In experiments on modular addition, a common benchmark for the grokking phenomenon, ReLU networks trained with a supported-Polar hidden-layer approach continue to make progress even when the loss curve appears to have plateaued.
This matters because grokking is typically marked by a sudden late jump in generalization after a long period of apparent overfitting. If training loss is flat during that period, standard monitoring based on loss alone would miss the ongoing internal changes. The title of the paper — "Flat Loss, Evolving Features" — captures the central claim: representations are still evolving even when the loss is not.
The findings are based on a single preprint and have not yet been peer-reviewed. Still, they offer a useful caution for practitioners: a flat training curve does not necessarily mean learning has stopped, and the features being learned may be preparing the network for a later breakthrough.