Centered kernel alignment (CKA) is widely used to compare neural network representations, but it has known shortcomings: it ignores the geometry of the underlying data manifold and relies on heuristics that make results vary with dataset scale. The authors of a new arXiv preprint propose Manifold-approximated Kernel Alignment (MKA), which explicitly incorporates local neighborhood structure into the alignment computation.

Rather than using representation-dependent degree weighting, MKA produces a closed-form alignment score and gives each row a probabilistic interpretation as a local Markov transition distribution. The authors also prove stability results for both the kernel and the score with respect to the choice of neighborhood size, addressing a common source of instability in manifold-based methods.

In controlled synthetic experiments, MKA was consistently less sensitive to hyperparameters and better captured topology-preserving relationships than existing approaches. On a large-scale benchmark, it achieved its strongest aggregate performance in vision, was competitive on graphs, and lagged behind CKA on several NLP tasks. The authors suggest that manifold-aware alignment could be useful whenever local neighborhood geometry matters, such as in representation learning.