Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM

vLLM, an open-source LLM inference engine, improves time-to-first-token (TTFT) and time-per-output-token (TPOT) by introducing a push-based KV cache transfer method using NVIDIA's NIXL over RDMA networks like AWS EFA. This method reduces latency during disaggregated prefill/decode inference by moving transfer coordination off the critical path, enhancing responsiveness especially under high load.