Transformer language models have been observed to develop attention heads that appear to track syntax, such as linking verbs to their subjects. This has been taken as evidence that the model encodes grammatical structure.
The new paper asks a sharper question: do these heads actually influence the model's predictions, or are they byproducts? The abstract poses this causal question directly.
The abstract begins to describe evidence, but the text cuts off. As a result, the article cannot report the study's conclusion. Readers will need to consult the full paper.