What Is Self-Attention? The Mechanism That Powers Transformers

Self-attention is a mechanism in AI transformers that enables each token in a sequence to create a context-aware representation by weighting information from other tokens. It involves a five-step process including projecting tokens, scoring relevance, normalizing, combining values, and repeating across multiple heads and layers.