A new preprint on arXiv asks a pointed question: could model-free reinforcement learning algorithms actually be learning a model of the environment? The authors propose to investigate this by looking inside the agent's value function, the learned signal that estimates how good a state or action is.
The method centers on counterfactual interventions: changing one targeted aspect of a state while holding everything else fixed. By observing how the value function responds to such a change, the researchers aim to visualize what the agent 'expects' about the world.
The abstract frames this as a way to probe the boundary between model-free and model-based learning. If value functions encode structured knowledge about environment dynamics, then model-free algorithms may be less model-free than they appear. The preprint's available abstract describes the motivation and approach, leaving the findings for the full paper.