In offline model-based reinforcement learning (MBRL), a policy is learned from pre-collected data with the help of a learned dynamics model. Choosing which dynamics model to use for downstream policy learning is critical, but standard approaches such as validation or off-policy evaluation are often inaccurate in this setting because of distribution shift. The authors of a new paper argue that this makes model selection a key bottleneck for offline MBRL.
To address this, they propose BOMS, an active model selection framework that treats model selection as a Bayesian optimization problem. BOMS uses a small online interaction budget to actively probe candidate models, and it enables probabilistic inference through a novel model-induced kernel. The kernel is designed to be both theoretically grounded and computationally efficient, allowing the framework to scale to practical RL tasks.
Experiments show that BOMS improves over baseline model selection methods on a variety of RL tasks, while requiring online interaction comparable to only 1–2.5% of the offline training data. This suggests that even a small amount of targeted online data can substantially improve model selection in offline MBRL, without undermining the offline setting's goal of avoiding large-scale environment interaction.