Valkey 9.2 Launches with Distributed KV Caching for AI Inference

Unite.AI
Read full story· 5 min
Valkey launched version 9.2 featuring distributed key-value caching for AI inference, reducing inference costs by up to 40% and first-token latency by up to 80% on existing GPU setups. The shared cache stores reusable tensors across GPU servers, improving efficiency for repeated context workloads without requiring application code changes.

What happened

Valkey released version 9.2 on November 19, 2026, introducing distributed key-value caching for AI inference workloads. The new feature allows GPU servers to share and reuse tensors generated during model processing, which can reduce both inference costs and response times. The update is available for self-managed deployments using open-weight models, and can be integrated with engines like vLLM and SGLang through a configuration change. The release also includes infrastructure improvements such as forkless snapshots, compression for replication, and memory optimizations for large sorted sets.

Key details

  • Valkey 9.2 was released on November 19, 2026.
  • Distributed key-value caching can cut inference costs by up to 40% and first-token latency by up to 80% on existing GPU setups.
  • The shared cache works with engines like vLLM and SGLang via the LMCache connector, requiring no changes to application code.
  • A new Path Hash data type enables efficient lookup and prefix matching in the cache.
  • Infrastructure updates include forkless snapshots, LZ4 and Zstandard compression for replication, and a switch to B+ trees for large sorted sets.

Why it matters

Operators running AI inference on GPUs can now serve more requests and reduce spending without changing their application code, as the cache enables reuse of computation across servers. Organizations with workloads that frequently repeat context, such as chat or tool use, can see significant reductions in both cost and response time. The infrastructure improvements also lower operational overhead for those managing large-scale Valkey deployments, especially where network and memory costs are a concern.


More in Machine Learning