Quantile distributional reinforcement learning aims to characterize the full distribution of discounted cumulative rewards under a policy, rather than only its expectation. A common approach is to represent the return distribution by a vector of quantiles, defined as the fixed point of a quantile-projected distributional Bellman equation. This paper by Cheng, Peng, and Zhang develops online statistical inference procedures for that quantile vector, allowing practitioners to construct confidence intervals while data arrive sequentially.

Two procedures are proposed. The first builds on the trajectory of quantile temporal-difference learning (QTD). The authors prove functional central limit theorems for the partial sums of QTD iterates and introduce a recursively computable random-scaling statistic; the resulting confidence intervals have asymptotically exact coverage at deterministic horizons, with length shrinking as T^{-1/2} in probability. The second procedure constructs anytime-valid confidence intervals directly from samples. It exploits the monotonicity of the quantile Bellman map to design a coordinate-shaving algorithm that provides finite-sample simultaneous coverage, and under positive limiting state-visit frequencies the interval length is of order sqrt(log log T / T). A matching lower bound shows this rate is classwise optimal.

Together, the two methods offer complementary guarantees. The first is tailored to fixed-horizon settings with asymptotic exactness, while the second provides anytime validity with finite-sample coverage. Both are developed for synchronous and asynchronous sampling from a generative model, as well as for Markovian data, making the results broadly applicable to online policy evaluation in reinforcement learning.