Activation probes are a common technique for detecting misbehavior in language models by reading their internal activation vectors. The method's validity rests on the assumption that models cannot deliberately alter what those activations encode.
A new arXiv preprint draws a parallel to human neurofeedback, where people learn to control their own brain activity. The authors investigate whether LLMs can similarly learn to control their activations in context, and the title indicates they find only limited control.
If models can influence their activations, probes may become less reliable as a window into internal states. The study's findings suggest that while some control is possible, it is constrained, so probes are not entirely invalid but should be used with caution.