Puzzles have long been a useful tool for probing AI reasoning because they combine verifiable goals with partial information. A new arXiv paper, 'Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving,' builds on this idea to evaluate large language models (LLMs) in a more interactive setting.
The paper's abstract notes that puzzles require multi-step decisions to reach a solution, making them a natural testbed for LLM reasoning. However, it argues that existing puzzle benchmarks are insufficient for this purpose. The authors propose a generalized puzzle-solving benchmark designed to assess multi-turn knowledge reasoning, where models must query for missing information and reason iteratively.
Because the abstract is truncated, the full details of the benchmark and any experimental results are not available in the source. Still, the framing suggests a move toward more dynamic evaluation of LLMs, where reasoning is measured not just on a single answer but on the ability to ask the right questions along the way.