Google Cloud, Red Hat, and IBM have released llm-d 1.0, an open-source, Kubernetes-native stack for distributed LLM inference. The project is designed to serve large models across fleets of TPUs and GPUs, with support for the vLLM and SGLang inference engines.
The new version adds multi-tenant flow control to isolate traffic between users or applications, and zero-downtime upgrades so the stack can be updated without interrupting running workloads. It also introduces agent-aware routing across vLLM, SGLang, TPUs, and GPUs, according to the announcement.
Because llm-d is Kubernetes-native, it can be managed with existing cluster tooling. The release is a collaboration between Google Cloud, Red Hat, and IBM, and is available as open-source software.