Researchers on arXiv compare transcoders with sparsified MLP layers in transformer language models. Transcoder features are often treated as more interpretable than raw MLP neurons, which are known to be polysemantic. The new analysis asks whether those features are actually distinct from the original neurons.

The authors report that many transcoder features closely match individual MLP neurons. In other words, the transcoder is not always discovering a cleaner decomposition; it is often relabeling the same polysemantic units. This raises questions about how much interpretability transcoders add over simpler sparsification methods.

The preprint is titled "Comparing Sparsified MLPs and Transcoders: Many Transcoder Features Are Neurons in Disguise." It has not yet appeared in a peer-reviewed venue, so the findings should be treated as preliminary.