Amazon Launches SageMaker HyperPod Inference Gateway to Enhance GPU Routing for LLMs
Covered by 2 sources
Amazon introduced SageMaker HyperPod Inference Gateway, a Kubernetes-native addon that uses real-time GPU metrics to optimize routing of large language model inference requests, reducing latency by up to 82% and improving GPU utilization without application changes.