Amazon SageMaker HyperPod adds NVIDIA Dynamo for multi-node LLM serving

Covered by 2 sources
Read onAWS Blog· 8 min
Amazon SageMaker HyperPod Inference now includes NVIDIA Dynamo, enabling managed distributed serving of large language models with improved throughput and latency. The integration uses HyperPod's GPU scheduling and networking features to optimize performance and resource use, doubling output-token throughput per GPU under AWS latency targets.

What happened

Amazon Web Services has integrated NVIDIA Dynamo, an open-source distributed inference framework, into Amazon SageMaker HyperPod Inference. This update allows SageMaker HyperPod to efficiently serve large language models, such as GLM-5.3, across multiple nodes by separating prefill and decode workloads. The integration uses HyperPod's GPU scheduling, Elastic Fabric Adapter networking, and node health monitoring to manage resources and improve inference performance. The change is now available for users deploying large language models on SageMaker HyperPod Inference.

Key details

  • NVIDIA Dynamo is now supported as a managed option in SageMaker HyperPod Inference.
  • The integration enables disaggregated serving of large language models, including GLM-5.3.
  • Benchmarks showed more than double the highest aggregated output-token throughput per GPU under AWS latency targets.
  • HyperPod leverages Elastic Fabric Adapter networking and coordinated GPU scheduling for performance.
  • Users can choose between aggregated or disaggregated serving, and select inference engines like vLLM, SGLang, or TensorRT-LLM.

Why it matters

Organizations running large language models on AWS SageMaker HyperPod can now serve models that exceed the memory of a single GPU instance by distributing workloads across multiple nodes. This reduces inference latency and increases throughput, allowing teams to process longer prompts and larger models more efficiently. The integration also simplifies resource management and scaling for machine learning teams deploying advanced language models.

Also covered by 1 source


More in Machine Learning