Tiered KV cache for large LLMs on Amazon SageMaker HyperPod with Curvine
AWS Blog

Amazon SageMaker HyperPod now supports a tiered KV cache for large language models, integrating GPU, CPU, and shared NVMe storage via Curvine to improve cache reuse and reduce time-to-first-token. This architecture boosts cache hit rates across replicas and cuts infrastructure costs by enabling use of smaller GPU instances.
