AWS Routes HyperPod Inference Across Regions From One Endpoint

Unite.AI
Read full story· 5 min
AWS launched Amazon SageMaker HyperPod Global Inference Gateway, enabling a single managed endpoint to route inference requests across multiple HyperPod clusters in one or more AWS Regions, optimizing capacity use and failover.

What happened

On November 20, 2026, AWS launched the Amazon SageMaker HyperPod Global Inference Gateway. This new service allows customers to send inference requests to a single managed endpoint, which then routes those requests across multiple HyperPod clusters, potentially in different AWS Regions. The system automatically selects a cluster based on capacity, queue depth, proximity, and recent error rates, and provides automatic failover if a cluster becomes unavailable. The launch adds a global routing tier above the existing cluster-level gateway, simplifying how clients access AI model serving infrastructure.

Key details

  • The service is available in 13 AWS Regions, including US, Europe, South America, and Asia Pacific locations.
  • Cluster selection uses four signals: available KV cache/GPU memory, queue depth, region proximity, and error rate, each with configurable weights.
  • Automatic failover shifts traffic to healthy clusters without client changes, and failed requests are retried once unless streaming has started.
  • Clusters are published to the gateway using spec.globalRouting in their configuration, and unpublished clusters cannot be added.

Why it matters

Companies running large-scale AI models on AWS can now manage inference across multiple clusters and regions from a single endpoint, reducing the need to build and maintain custom load balancing and cross-region networking. This change can lower operational overhead for platform teams and improve resource utilization by routing requests to clusters with available capacity. Organizations can achieve cross-region failover and capacity overflow without changing client code, which can help maintain service availability and performance during traffic spikes or outages.

More on this story

  1. Keep inference running through regional capacity shortfalls with cross-Region inference routing in Amazon SageMaker AI
    · 2 sources
  2. How Salesforce is advancing global-scale model serving with cross-Region inference routing in Amazon SageMaker AI
    · AWS Blog

More in Chips & Compute