A new paper on arXiv examines a property shared by many widely used activation functions, including ReLU, GELU, Swish, and Sigmoid. Because these functions produce predominantly positive outputs, the average post-activation value is not zero. The authors call this phenomenon "activation bias."

The paper argues that activation bias is common across neural networks, yet its effects are rarely studied. The abstract suggests that reducing this bias can improve learning, implying that the nonzero mean may be a hidden drag on training dynamics.

The full details of the proposed method and experiments are not included in the abstract, so the specific mechanism remains unclear. Still, the claim points to a simple design consideration that could matter for practitioners choosing activation functions.