The PATTERN benchmark, a synthetic node-classification task still used to rank graph networks, may be far easier than intended. The benchmark works by planting a random subgraph in a random host graph and asking which nodes belong to it. But the generator draws 100 patterns once and plants the same ones over and over, meaning the exact target subgraphs recur across test instances.

A new arXiv preprint argues that this design flaw lets models solve PATTERN from memory. The authors propose generator-aware baselines that exploit the reused patterns, showing that high performance on the benchmark does not necessarily reflect genuine structural understanding.

The finding does not contradict the original benchmark's description, but it highlights a serious limitation. If the same patterns are recycled, then ranking graph networks on PATTERN may reward memorisation rather than generalisation. The authors suggest that generator-aware baselines should be included in future comparisons to account for this effect.