Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference

Amazon SageMaker Inference has launched prefix-aware routing, a method that routes requests with identical prompt prefixes to the same instance, enabling effective reuse of cached computations. This approach significantly reduces latency and boosts throughput for large language models like Llama 3.1 70B by increasing cache hit rates.