Choosing the scale of weight initialization is a classic practical problem in deep learning, and widely used schemes such as He-normal initialization are largely heuristic. In a revised arXiv paper, Hiroshi Horii and Sothea Has propose a criterion derived from the dynamics of stochastic gradient descent itself. They model SGD as a continuous-time Langevin process governed by a Fokker-Planck equation, then study the KL divergence between the quasi-stationary distribution of the parameters and their initial distribution.

Because the quasi-stationary distribution depends on the expected cost function, this divergence lets the authors bound the expected loss explicitly in terms of initialization parameters. Minimizing that bound yields an optimal condition for the variance of Gaussian weight initialization. The result gives a concrete mathematical rule rather than a tuning heuristic, and the authors argue it clarifies the physical meaning of parameter dynamics in deep networks.

The authors test their criterion by training fully connected networks with classical SGD on MNIST and Fashion-MNIST. According to the paper, when the initialization variance satisfies their condition, the models consistently achieve lower final training loss and higher test accuracy than models using conventional He-normal initialization. The theoretical result is derived for the Gaussian case, and the experimental confirmation is limited to the architectures and datasets reported in the paper.