AWS Routes HyperPod Inference Across Regions From One Endpoint

What happened
On November 20, 2026, AWS launched the Amazon SageMaker HyperPod Global Inference Gateway. This new service allows customers to send inference requests to a single managed endpoint, which then routes those requests across multiple HyperPod clusters, potentially in different AWS Regions. The system automatically selects a cluster based on capacity, queue depth, proximity, and recent error rates, and provides automatic failover if a cluster becomes unavailable. The launch adds a global routing tier above the existing cluster-level gateway, simplifying how clients access AI model serving infrastructure.
Key details
- The service is available in 13 AWS Regions, including US, Europe, South America, and Asia Pacific locations.
- Cluster selection uses four signals: available KV cache/GPU memory, queue depth, region proximity, and error rate, each with configurable weights.
- Automatic failover shifts traffic to healthy clusters without client changes, and failed requests are retried once unless streaming has started.
- Clusters are published to the gateway using spec.globalRouting in their configuration, and unpublished clusters cannot be added.
Why it matters
Companies running large-scale AI models on AWS can now manage inference across multiple clusters and regions from a single endpoint, reducing the need to build and maintain custom load balancing and cross-region networking. This change can lower operational overhead for platform teams and improve resource utilization by routing requests to clusters with available capacity. Organizations can achieve cross-region failover and capacity overflow without changing client code, which can help maintain service availability and performance during traffic spikes or outages.
