Reinforcement learning usually optimizes expected reward, but many real-world objectives are nonlinear functions of the state-action distribution. This paper addresses that broader problem, called reinforcement learning with general utilities (RLGU), where the goal is to maximize a concave functional of the occupancy measure induced by the policy.
The authors develop a natural actor-critic algorithm for concave RLGU objectives. A central contribution is the observation that these objectives have an exact variational dual, which the method leverages. This dual formulation is what allows the algorithm to achieve sample-efficient learning, according to the paper.
The method is described as using neural dual critics, indicating a practical deep-learning implementation. The abstract does not include experimental details, so the reported efficiency is based on the theoretical framing rather than benchmark results.