LLM inference servers hold a key-value cache for each active request, and that cache grows with every generated token. A new paper models this as a dynamical system and shows that the growth itself can become a source of congestion: under high concurrency, aggregate memory use rises endogenously, forcing the system to evict and restart active requests, which wastes computation and reduces throughput.

Using a discrete-time model of continuous batching, the authors analyze admission, memory growth, and eviction. For homogeneous workloads, they prove that the eviction-free equilibrium is unstable and that, except for a measure-zero set of exact captures, the system converges to a unique worst-case limit cycle with evictions, losing up to 50% of throughput. For heterogeneous workloads, they derive a stability criterion for two classes and show how a survival-polynomial mechanism generalizes to multiple classes.

In an input-dominated scaling regime, coprime decoding lengths stabilize the eviction-free equilibrium, while non-coprime lengths create synchronized modes that drive instability. The paper's v2 adds GPU experiments with benchmark workloads. The authors frame service-induced congestion as a structural instability mechanism and propose scheduling design principles to sustain high throughput.