Multi-hop knowledge-graph question answering (KGQA) benchmarks are commonly viewed as stable tests of language understanding and reasoning. But a new arXiv preprint argues that this view misses a key variable: the graph itself. For methods that navigate the knowledge graph to find answers, the same question can be easier or harder depending on how the graph is built.
The authors conduct a controlled audit of two widely used benchmarks, PathQuestion and PathQuestion-Large, to separate the effect of graph construction from the effect of the question wording. Their findings suggest that benchmark difficulty is not purely a property of the task as written, but emerges from the interaction between the question and the graph structure.
This has practical implications for comparing KGQA systems. If graph construction varies across benchmarks or versions, then performance differences may reflect graph topology rather than reasoning ability. The paper calls for treating graph construction as a first-class factor in benchmark design and evaluation.