Concurrent HBM and Host Memory Access Speeds LLM Inference

Large language models often need more memory than a GPU's high-bandwidth memory (HBM) can provide, forcing a tradeoff between speed and capacity. A new runtime system called BOOST, developed by researchers at Georgia Tech, Nvidia Research, and Stanford University, aims to remove that tradeoff by allowing the GPU to access host memory and HBM at the same time.

The system is described in a technical paper titled "BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inference." The authors position BOOST as the first runtime system to provide this kind of concurrent access, which could improve inference throughput by reducing the time GPUs spend waiting for data that does not fit in HBM.

The paper's abstract highlights the core idea: instead of treating host memory as a slow fallback, BOOST overlaps accesses to both memory types so that the GPU can keep working while data moves. This approach could help deploy larger models on existing hardware without requiring additional HBM capacity.