Google Cloud Ships llm-d 1.0 With Cache-Aware Routing for TPU and GPU Fleets

Google Cloud released llm-d 1.0, a Kubernetes-native distributed inference stack that coordinates vLLM and SGLang model servers for improved performance on TPU and GPU fleets. It features cache-aware routing and multi-tenant quality of service, boosting throughput and reducing latency in large-scale deployments.