Large language models have shown impressive results across a wide range of tasks, but their logical reasoning abilities remain uneven. A new arXiv paper, “Competence-Based Evaluation of Logical Reasoning in Large Language Models,” points to persistent weaknesses, including inconsistencies that emerge when the same reasoning problem is expressed with different linguistic variations.

The paper’s title signals a focus on competence-based evaluation, suggesting a shift from task-level accuracy toward a more structured assessment of underlying reasoning ability. The abstract indicates the work is aimed at improving this area, though the released text is truncated before the full method is described.

As released, the abstract offers limited detail, but it clearly identifies linguistic variation as a key stress test for LLM reasoning. The proposed evaluation approach appears designed to expose such inconsistencies rather than relying on aggregate performance scores alone.