Google Open-Sources TPU Sync for KV Cache and Weight Transfer on TPUs

What happened

Google Cloud has released TPU Sync, an open-source C++ library designed to speed up the transfer of KV cache blocks and model weights on TPUs. The tool is now available on GitHub under the Apache 2.0 license and supports both JAX and PyTorch frameworks. TPU Sync targets workloads such as long-context large language model serving and reinforcement learning, where fast data movement is critical. The release coincided with the announcement of new TPU 8tx and TPU 8i Max systems.

Key details

• TPU Sync reduces KV cache transfer latency by up to 90% for large models, such as Qwen3 235B and a 480B model.
• The library provides Python wheels for JAX and PyTorch, but only as development builds, and recommends pinning to a stable commit.
• TPU Sync supports KV cache management, KV store, and weight synchronization through its APIs.
• It is validated on TPU v7x and v6e hardware, and the PyTorch wheel requires a specific torch_tpu build.
• The repository includes architecture docs, C++ and Python APIs, and examples for disaggregated serving and KV offloading.

Why it matters

Developers and organizations running large language models or reinforcement learning workloads on TPUs can now move data between memory tiers much faster, reducing wait times and improving efficiency. This change can lower operational costs and enable more responsive AI services, especially for applications that require handling long contexts or frequent model updates. Users must be aware that the APIs are still evolving and that only development builds are currently available, so stability and compatibility should be considered before adoption.