Amazon SageMaker HyperPod adds NVIDIA Dynamo for multi-node LLM serving

What happened
Amazon Web Services has integrated NVIDIA Dynamo, an open-source distributed inference framework, into Amazon SageMaker HyperPod Inference. This update allows SageMaker HyperPod to efficiently serve large language models, such as GLM-5.3, across multiple nodes by separating prefill and decode workloads. The integration uses HyperPod's GPU scheduling, Elastic Fabric Adapter networking, and node health monitoring to manage resources and improve inference performance. The change is now available for users deploying large language models on SageMaker HyperPod Inference.
Key details
- NVIDIA Dynamo is now supported as a managed option in SageMaker HyperPod Inference.
- The integration enables disaggregated serving of large language models, including GLM-5.3.
- Benchmarks showed more than double the highest aggregated output-token throughput per GPU under AWS latency targets.
- HyperPod leverages Elastic Fabric Adapter networking and coordinated GPU scheduling for performance.
- Users can choose between aggregated or disaggregated serving, and select inference engines like vLLM, SGLang, or TensorRT-LLM.
Why it matters
Organizations running large language models on AWS SageMaker HyperPod can now serve models that exceed the memory of a single GPU instance by distributing workloads across multiple nodes. This reduces inference latency and increases throughput, allowing teams to process longer prompts and larger models more efficiently. The integration also simplifies resource management and scaling for machine learning teams deploying advanced language models.

