Offline reinforcement learning aims to find optimal policies from pre-collected data, but standard methods often assume the data comes from a single episode or homogeneous batch episodes. That assumption breaks down when the population is heterogeneous: a single policy learned for everyone can be suboptimal for many individuals.
Miao, Shahbaba, and Qu propose an individualized offline policy optimization framework for heterogeneous time-stationary Markov decision processes. Their model introduces individual latent variables to efficiently estimate individual Q-functions, and their Penalized Pessimistic Personalized Policy Learning (P4L) algorithm guarantees a fast rate on average regret under a weak partial coverage assumption on behavior policies.
In simulation studies and a real-data application, P4L showed superior numerical performance compared with existing methods. The paper is an arXiv preprint (2505.09496), last revised in October 2026. Since only one source was provided, no independent comparison is possible.