Sparse autoencoders (SAEs) are used to break down large language model activations into sparse dictionary atoms, with the goal that each distinct concept maps to its own feature. However, a recurring behavior called feature absorption undermines this clean separation: a parent concept can absorb or overshadow related features, making the representation less interpretable than intended.
HiPACE, introduced in a new arXiv paper, targets this problem directly. The method performs hierarchical phase-boundary analysis to detect where and how absorption occurs, and then applies a controlled evaluation framework to measure its impact. This allows researchers to see the boundaries of feature absorption and assess whether a feature is truly independent or being subsumed by a broader concept.
The authors frame HiPACE as a way to make SAE-based interpretability more robust. By identifying and controlling absorption, the method could help ensure that features remain distinct and useful for downstream analysis. While the abstract does not provide experimental results, the approach suggests a systematic path toward addressing one of the key challenges in sparse autoencoder interpretability.