Offline reinforcement learning is constrained by the coverage of its fixed dataset, while online exploration can be costly or unsafe. This paper asks whether the learner can get around that limitation by querying a collaborator—such as a simulation service—rather than interacting with the real environment. The learner picks state-time pairs, the collaborator returns an action from a fixed policy, and the simulator reveals the resulting reward and next state.
The authors formalize this as query-based collaborative reinforcement learning and study finite-horizon tabular problems. They derive an instance-dependent information-theoretic lower bound on the number of additional queries needed to identify an epsilon-optimal policy, and they use a local quadratic approximation to characterize that bound. Based on this result, they design an adaptive query-allocation algorithm whose query complexity nearly matches the local benchmark under suitable conditions.
Experiments on multiple synthetic and semi-synthetic environments show that the proposed method is empirically efficient, suggesting that targeted queries can meaningfully compensate for sparse offline data without requiring direct environment access. The paper is a single-source work, so there are no conflicting findings to compare.