Large language models generate text autoregressively, predicting one token at a time. To avoid reprocessing the entire context with every new token, inference frameworks use KV caching, which stores intermediate results from earlier tokens. This splits inference into two phases: prefill, which processes the full prompt and builds the cache, and decode, which generates subsequent tokens using cached data.
These phases stress hardware differently. Prefill relies on large matrix-matrix multiplications and tends to be compute-bound, while decode uses matrix-vector multiplications and is often limited by memory bandwidth. The two key user-facing metrics correspond to these phases: Time to First Token (TTFT) measures how long the prefill stage takes before the first output appears, and Inter-Token Latency (ITL) measures the time between generated tokens during decode.
Because the operations in both phases are highly parallel, GPUs are well suited to LLM inference. The article notes that even in power-constrained edge devices, GPU architectures like Imagination's PowerVR can accelerate the large GEMM operations in prefill and keep decode responsive. The distinction between TTFT and ITL matters for deployment decisions, especially in interactive applications where a slow first response or sluggish token generation degrades user experience.