What exactly is a token?
A language model never reads letters. Before any text reaches it, a tokenizer chops the text into tokens, where a token is one entry in a fixed list called the model's vocabulary. It is the smallest unit the model reads and writes: not a word, not a character, not a syllable, but whatever unit the tokenizer settled on, usually a subword. What the model actually receives is never the characters themselves, but a list of integer IDs, one per token, where each ID is that token's position in the vocabulary.
A subword is a piece bigger than a single letter but often smaller than a whole word. A common word like "the" is typically one token on its own, while a longer or rarer word gets split into several pieces, so "tokenization" becomes "token" + "ization."
Sign up free to read, listen and talk to tutor
Want me to explain it differently?
AI concepts can be dense. Tell me what's confusing and I'll find a new analogy.