Most research on efficient large-language-model inference has focused on cutting the cost of decoding once a full request has been received. A new preprint on arXiv argues that this misses what users actually experience: the time between submitting a request and getting an answer. The authors propose streaming generation as a way to optimize that "time-to-answer" directly.
According to the abstract, the paper's starting point is that the quantity a user experiences is the interval between submitting a request and the answer arriving. Rather than treating decoding as a post-request cost to be minimized, the work frames generation as something that can begin and stream output while the user is still typing.
Because the abstract is the only portion available in the source, the specific method and experimental results are not described here. The contribution, as stated, is to shift the optimization target from decoding cost to user-perceived latency.