What Is Tokenization? How AI Turns Text Into Tokens

Tokenization is the process of converting raw text into discrete units called tokens, which AI models map to identifiers for mathematical processing. This involves five key steps: normalizing input, splitting into pieces, mapping to integers, adding controls, and decoding back to text. Understanding tokenization precisely is crucial as it defines a clear operational boundary distinct from simpler text splitting methods.