Cut self-hosted LLM cost and latency with the distributed KV caching agent skill

AWS Blog
Read full story· 8 min
Amazon Web Services introduced a distributed key-value caching skill for self-hosted large language models that reduces GPU inference costs by up to 40% and speeds up response times by up to 80%. The skill integrates with AWS Agent Toolkit and supports multiple coding agents, enabling efficient reuse of KV attention states across sessions and servers.

What happened

Amazon Web Services has launched a distributed key-value caching skill for users running large language models on their own infrastructure. The new feature is part of the AWS Agent Toolkit and works with several coding agents to help reuse key-value attention states across different servers and sessions. AWS says this reduces the need for repeated GPU computations, cutting costs and response times for self-hosted LLMs. The skill is available now and integrates with existing tools like Amazon ElastiCache.

Key details

  • Reduces GPU inference costs by up to 40%
  • Speeds up response times by up to 80%
  • Integrates with AWS Agent Toolkit and supports agents like Kiro and Claude Code

Why it matters

Companies and developers who self-host large language models can now lower their operational costs and improve user experience by minimizing delays in AI responses. This change allows teams to reuse cached data across GPU servers, reducing redundant computations and making it more practical to handle long, multi-turn conversations or coding tasks. Organizations using AWS infrastructure and compatible coding agents can adopt this feature to optimize their AI workloads without significant changes to their existing setup.


More in Dev