Amazon SageMaker AI adds KV cache-aware routing to speed real-time endpoints

Covered by 2 sources
Read onAWS Blog· 21 min
Amazon SageMaker AI now routes requests to instances with relevant GPU cache data, boosting performance for large language models handling long or multi-turn prompts. This update improves median time to first token by up to 97% and throughput by 234% in tests.

What happened

Amazon Web Services introduced KV cache-aware routing for SageMaker AI real-time endpoints. This new feature directs incoming model requests to the instance that holds the most relevant cached data, based on the current state of each instance's GPU cache. The update is designed for applications using large language models, especially those with ongoing conversations or long prompts. Users can enable this routing by setting an environment variable in their model and updating the endpoint configuration.

Key details

  • KV cache-aware routing improves median time to first token by up to 97% in long-prompt tests.
  • Throughput can increase by up to 234% when using this feature.
  • The routing uses real-time cache state reported by each model container.
  • It is available only when at least two instances or inference component copies are running.
  • Users enable it by setting an environment variable and selecting 'KV_AWARE' as the routing strategy.

Why it matters

Teams running large language model applications on SageMaker AI can now handle multi-turn conversations and long prompts more efficiently. This reduces response times and boosts throughput, which can lower operational costs and improve user experience for services like chat assistants and retrieval-augmented generation systems. Organizations must update their endpoint configurations and model containers to use this feature.

Also covered by 1 source


More in Machine Learning