A new paper on arXiv argues that most interpretability tools for transformers miss a basic question: does a representation keep its meaning when the same content is phrased differently? The authors propose treating paraphrase pairs as local witnesses—pairs of activations that should agree because they carry the same underlying content.
From these local agreements, the paper builds a cellular decomposition intended to glue local contexts into global meaning. The title describes this as a sheaf-theoretic decomposition of transformer representations, suggesting a mathematical structure for tracking how meaning is assembled across a model's internal states.
The abstract is brief, so the full construction and experimental results are not yet available from the source. As a single-source digest, this summary reflects only what the authors state in their abstract.