Tiered KV cache for large LLMs on Amazon SageMaker HyperPod with Curvine

AWS Blog
Read full story· 29 min
Amazon SageMaker HyperPod now supports a tiered KV cache for large language models, integrating GPU, CPU, and shared NVMe storage via Curvine to improve cache reuse and reduce time-to-first-token. This architecture boosts cache hit rates across replicas and cuts infrastructure costs by enabling use of smaller GPU instances.


More in Dev