A new paper by Johan Jonasson extends earlier work on noise sensitivity in fully connected deep neural networks with sign activations. The earlier result, with Jonasson, Steif and Zetterqvist, showed that networks with i.i.d. standard Gaussian weights and no bias terms are both annealed and quenched noise sensitive under a mild positive-correlation condition on the input and its perturbation, provided the perturbation is asymptotically unlikely to be exactly the original input or its negative.
The new work adds independent Gaussian biases of the natural order. It proves that quenched noise sensitivity survives over a broad range of depths under a slightly weaker assumption: only that the perturbed input is asymptotically unlikely to coincide with the original input, not with its negative. Combined with the positive-correlation condition, this is the minimal requirement for the perturbed input to be a genuine perturbation.
Interestingly, the annealed covariance behaves differently. It remains bounded away from zero, and the paper shows this persistent covariance is entirely due to fluctuations in the network-dependent output means. So while individual typical networks are sensitive to noise in the quenched sense, the averaged correlation does not vanish because of variability across networks.