Google claims its new AI software cuts inference latency by over 90% — but there’s a catch

Google introduced TPU Sync, an open-source system to accelerate AI inference by reducing KV-cache transfer latency by up to 90% on large models with long context windows, improving time to first token and throughput. The software optimizes data movement across memory types using zero-copy transfers, but results vary by workload and network bandwidth. Nvidia's systems still outperform TPUs in some inference scenarios.