Partially observable Markov decision processes remain hard for model-based deep RL because an agent must infer hidden state, learn dynamics, and plan under beliefs. In practice, these components are typically glued together with approximations whose errors are poorly understood, while classic POMDP theory tends to study algorithms too computationally heavy for modern RL. This paper proposes an algorithm that tackles all three pieces at once: a sequential Monte Carlo particle filter estimates beliefs, a latent dynamics model is updated online, and planning runs d steps ahead before bootstrapping with a QMDP state-value critic.
The main contribution is an end-to-end regret bound that decomposes total suboptimality into model estimation error, particle filter error, Monte Carlo rollout error, and a structural bias term from the QMDP approximation. The bound makes explicit how statistical sample size, particle budget, rollout budget, and planning depth affect performance; in particular, deeper planning shrinks the structural bias. This gives practitioners a theoretical handle on trade-offs that are often made implicitly.
The authors instantiate the algorithm in an MPC-based deep RL framework and test it on a variety of environments. On a POMDP constructed to satisfy the assumptions, each term in the bound decays at its predicted rate. The paper is a single-source report, so these results are presented as the authors' claims rather than as an independent comparison.