Zyphra AI Introduces Zyphra Hierarchical Attention (ZHA): A Path to True Subquadratic Attention

Zyphra has launched Zyphra Hierarchical Attention (ZHA), a sparse attention method using a tree-structured indexer that enables efficient long-context processing up to 256K tokens with minimal parameter increase. ZHA retrofits frozen pretrained models like Qwen3.6-35B-A3B, achieving near dense attention performance on reasoning and coding benchmarks while significantly reducing indexing overhead.