Amazon SageMaker AI adds cross-region routing to handle GPU shortages

Covered by 2 sources
Read onAWS Blog· 17 min
Amazon SageMaker AI introduces cross-region inference routing that automatically directs requests to healthy endpoints across AWS Regions and accounts, improving availability during local GPU capacity shortages without changing existing deployments.

What happened

Amazon Web Services has launched cross-region inference routing for its SageMaker AI service. This new feature allows applications to automatically direct inference requests to healthy endpoints in different AWS Regions and accounts when local GPU capacity is insufficient. The routing is managed by a new resource called the inference router, which does not require changes to existing model deployments. The feature became generally available in June 2024.

Key details

  • Cross-region inference routing is now generally available in Amazon SageMaker AI.
  • The inference router automatically selects healthy endpoints across Regions and accounts.
  • Routing decisions are based on health signals such as server errors, throttling, and timeouts.
  • No changes are needed to existing SageMaker model deployments or scaling setups.
  • The router publishes health status to Amazon CloudWatch and manages failover without user intervention.

Why it matters

Organizations using SageMaker AI for model inference can now maintain higher availability during local GPU shortages without building custom routing solutions. This reduces operational complexity and helps avoid downtime or throttling when a Region runs out of capacity. Users benefit from uninterrupted service and do not need to modify their current deployment processes.

Also covered by 1 source

More on this story

  1. How Salesforce is advancing global-scale model serving with cross-Region inference routing in Amazon SageMaker AI
    · AWS Blog

More in Machine Learning