A Team of Researchers from Columbia and CertiK Introduce TensorLean for Formal Verification of GPU Kernels

What happened
Researchers from Columbia University and CertiK released TensorLean, a new tool for formally verifying GPU kernels at the PTX level. The team used TensorLean to check 149 different kernel instances, including all kernels from FlashAttention-2, -3, and -4, as well as DeepGEMM. Out of these, 83 kernel instances were proven correct according to their specifications. The process also uncovered 38 bugs, including a data race in FlashAttention-4 that had not been detected by previous testing.
Key details
- TensorLean is a Lean-based verifier that checks PTX-level GPU kernels against tensor-level math specifications.
- The tool analyzed 149 kernel instances and proved 83 correct.
- 38 bugs were found across 6 projects, with 23 in FlashAttention-2, -3, and -4.
- The verification process identified 19 memory access errors, 13 numerical bugs, and 7 data races.
- TensorLean uses symbolic execution and SMT solvers like Z3 and cvc5 to discharge proofs.
Why it matters
Developers and organizations using GPU kernels, especially those relying on FlashAttention and DeepGEMM, now have a way to formally verify correctness and catch subtle bugs that may not appear in standard testing. This can reduce the risk of undetected errors in high-performance computing and AI workloads, potentially saving time and resources spent on debugging and improving reliability. The discovery of previously unknown bugs means teams may need to review and update affected code to ensure correctness and safety.
