An arXiv preprint, "Deep lifts: Neural networks can generalize even with the dumbest optimizer," considers what happens when a parameter vector is drawn uniformly at random from the set of all weights that interpolate a given labeled dataset within a ReLU architecture. The abstract describes this setup, focusing on width-W, depth-D networks with continuous weights.

The title indicates the central finding: such random draws generalize well, meaning even the simplest possible optimizer—uniform random search—can succeed. This points to the possibility that the inductive bias for generalization is already encoded in the architecture and the data, rather than being introduced by sophisticated optimization algorithms.

If the result holds, it would shift attention from optimizer design to the geometry of the solution space. The abstract is brief, but the framing suggests that most interpolating parameters may also be good at generalizing, making the choice of optimizer less critical than the structure of the network and the training set.