Tokens and tokenization

How raw text becomes the numbers a language model actually reads. Defines a token as an entry in a fixed vocabulary, explains why models use subword tokens instead of words or characters, how a tokenizer is trained (byte-pair encoding and the unigram method), and how text turns into integer IDs. Then it works through the consequences a builder feels every day: why spaces and capitalization change tokens, why there is no "unknown word," why the same sentence costs more tokens in some languages, why spelling and counting letters are hard, and how token counts drive context limits, API cost, and latency. Ends with why the tokenizer is frozen to the model and how to inspect exactly how your own text tokenizes.