The paper asks what makes a deep network trainable and what its loss surface looks like. The authors propose a single construction using a polynomial activation f(z)=z+λz³, then clamp and project it onto the unit sphere. This simple choice lets them analyze the loss surface directly.
The construction operates at two scales. At large depth, it explains trainability; at small network size, it makes the loss surface solvable. This dual view connects the abstract geometry of the loss surface to practical questions of initialization and training.
The findings suggest that the activation's form dictates how networks should be designed, initialized, and trained. Because the activation is polynomial, the loss surface can be studied analytically, offering concrete guidance rather than heuristic rules.