A new arXiv preprint argues that benchmarks for large language models in classical planning often conflate two distinct sources of difficulty: the structural complexity of the planning problem itself and the semantic interpretability of the symbolic vocabulary used to represent it. The authors note that LLMs generally struggle with planning, but existing benchmarks rarely separate these factors.
The study examines seven STRIPS domains using BFS-optimal plans, presumably to isolate structural properties such as plan length or branching factor from the ease of mapping symbols to meanings. By drawing this distinction, the work could help researchers determine whether LLM failures stem from genuine reasoning limitations or from difficulty in grounding abstract symbols.
As the abstract is truncated, specific findings are not yet available. The contribution so far is a framework for disentangling these two dimensions, which may lead to more diagnostic planning benchmarks for LLMs.