A new arXiv paper examines layer normalization (LN) and root mean square normalization (RMSNorm). The authors note that both depend on their inputs and remain nonlinear at inference, so inserting either next to an activation may change the function a network represents.

The paper, titled "Activation-Adjacent Layer Normalization: Effects on Approximation Power," studies these effects. The framing implies that the placement of normalization relative to activations is not neutral, and may bear on a network's approximation power.

The abstract in the source cuts off after stating the research question, so the paper's findings are not described. The motivation alone highlights a potential interaction between normalization and activation that could matter for network design.