What is Tokenization?

Tokenization is the first step in almost every NLP pipeline. Before a language model can process text, the text needs to be converted into a format the model can work with. Tokenization splits raw text into discrete units called tokens, and then those tokens are mapped to numerical IDs that the model actually processes.

What constitutes a token depends on the tokenization scheme. Word tokenization splits text at spaces and punctuation: 'Hello, world!' becomes ['Hello', ',', 'world', '!']. Character tokenization treats each character as a token. Subword tokenization - used by most modern models including GPT and BERT - finds a middle ground, breaking uncommon words into meaningful pieces while keeping common words whole. The word 'tokenization' might become ['token', 'ization'].

Subword tokenization is particularly important for handling rare words, technical jargon, and misspellings. Instead of treating an unknown word as a single unknown token, the model can build up its meaning from familiar subword pieces. This dramatically improves a model's ability to handle text 'in the wild' with all its real-world variation.

The concept of a token is also important for understanding the cost and capacity of language models. Most API pricing for large language models charges per token, and models have a maximum context window measured in tokens. A rough rule of thumb for English text is that one token is approximately 0.75 words, though this varies by language and tokenization scheme.

Tokenization choices can have subtle but significant effects on model performance. Languages with complex morphology (like Finnish or Turkish) require different tokenization strategies than English. Code requires tokenization that preserves syntax. Multilingual models need tokenizers that handle dozens of languages fairly, which is a non-trivial engineering challenge in building globally useful language models.