Tokenization and embeddings

Concept#

A model cannot read letters. It computes on numbers. Tokenization is the step that turns text into a sequence of integer IDs, and it happens before anything in How LLMs work runs.

Modern tokenizersTokenizerThe component that splits text into tokens and maps them to integer ids, and back. Each model family has its own, so count tokens with the tokenizer of the model you call.Open the glossary entryClose split text into subword pieces. The most common recipe is Byte-Pair Encoding (BPE)Byte-pair encoding (BPE)A way to build a subword vocabulary by repeatedly merging the most frequent adjacent pair of symbols. Any text, even an unseen word, can then be written from a fixed set of pieces.Open the glossary entryClose (a data-compression algorithm that Sennrich et al., 2015 adapted to handle rare words in machine translation); SentencePiece and WordPiece are related families. Each token ID is then looked up in a learned embedding tableEmbedding tableThe learned matrix with one row per token in the vocabulary. Looking up a token id returns its row, and the table is trained together with the rest of the model.Open the glossary entryClose, which returns a dense vector. Position informationPositional encodingExtra information that tells the model where each token sits in the sequence. Attention alone has no sense of order, so position must be injected.Open the glossary entryClose is added, and that sequence of vectors is the input to the transformer stack.

Why It Matters#

Everything you pay for and everything that limits you is counted in tokens: price, context window, rate limits, and latency. If you estimate in words or characters you will be wrong, sometimes by a lot, and the error is not the same for every language. Interviewers ask “how does the model handle a word it has never seen” and “why is token count not word count” because the answers show whether you understand the system underneath the API.

Mental Model#

Tokenization is a LEGO set. The set holds a fixed number of pieces, from single bytes up to whole common words. Any text can be built from those pieces, even a word that never appeared in training, because worst case you build it from small pieces. Common words come as one big brick. Rare words, other scripts and unusual strings come as many small bricks, and you pay per brick.

Example#

The word “unbelievable” might split into ["un", "believ", "able"]. The exact split depends on the tokenizer, so treat that as illustrative. A rare proper noun may split into many small pieces. Text in some non-English scripts needs many more tokens than the same sentence in English. Petrov et al. (2023) found that the same text translated into different languages can differ in tokenized length by up to 15 times in some cases.

Technical Deep Dive#

• BPE training: start from bytes or characters, repeatedly merge the most frequent adjacent pair in a training corpus, and record each merge. The final merge list is the vocabularyVocabularyThe fixed set of tokens a model can read and produce. A bigger vocabulary gives fewer tokens per text but a larger embedding table and output layer.Open the glossary entryClose. At run time the tokenizer applies the merges to new text, so unseen words fall back to smaller pieces.
• Different models, different vocabularies: each model family trains its own tokenizer. Examples from public sources: OpenAI’s tiktoken defines gpt2 with 50,257 tokens, cl100k_base with a top token ID near 100k, and o200k_base near 200k (tiktoken source, checked 2026-10-09). Meta’s Llama 3 uses a 128K-token vocabulary and reports up to 15% fewer tokens than Llama 2 (Meta, 2024-04-18). Newer models trend toward larger vocabularies, but sizes vary and change, so always check the model you use.
• Vocabulary-size trade-off: a bigger vocabulary means fewer tokens per text (cheaper and shorter sequences), but a bigger embedding table and a bigger output layer.
• Embeddings are learned: the embedding table is a matrix with one row per token. Its rows are trained jointly with the rest of the model, they are not hand-designed. The vector length is a model-specific number, typically in the thousands.
• Numbers and code: digits and symbols may be split in ways that do not line up with place value or syntax. That is a main reason models make spelling and arithmetic slips, and why tools (code or a calculator) are the robust fix.

How big is the table? Illustrative arithmetic#

A 128,000-token vocabulary with 4,096-dimension vectors has 128,000 x 4,096 = 524,288,000 values in the embedding table alone. These numbers are illustrative, not a specific model. The point is that vocabulary size shows up directly in model size.

Diagram#

Code#

# Counting with the tokenizer that matches your model (OpenAI-family example).
import tiktoken

enc = tiktoken.get_encoding("o200k_base")   # pick the encoding for YOUR model
text = "unbelievable"
ids = enc.encode(text)
print(len(ids), [enc.decode([i]) for i in ids])   # run it: the split is tokenizer-specific

The same string through another model family’s tokenizer can give a different count. For billing and context budgets, use the tokenizer or token-counting tool your provider documents for that exact model (check the provider’s docs), and compare with the usage numbers returned in real responses.

Real-World Application#

Token counting drives cost estimation and context-window budgeting in every production LLM app. Teams pricing a feature must account for non-English users often paying more tokens, and so more cost and latency, for the same amount of text. A common safe workflow: sample real traffic in each language you support, count with the production model’s tokenizer, and budget from those measurements instead of a global “4 characters per token” rule of thumb. When you switch models, re-measure, because the tokenizer can change too. For example, Anthropic’s Claude Sonnet 5.5 migration guide says the same text produces about 30% more tokens than on Claude Sonnet 4.6, depending on content, and tells you to recount and revisit max_tokens and cost (Anthropic docs, checked 2026-10-09).

Interview Perspective#

Expect: “difference between a token and a word”, “why does token count matter for cost”, “explain BPE at a high level”. The mistakes to avoid are assuming 1 token = 1 word, or that tokenization is language-neutral.

Provenance: Common industry pattern. Standard topics for ML and GenAI roles, not a verified list for any named company.

Practice Questions#

• Basic: What problem does subword tokenization solve that word-level tokenization does not?
• Intermediate: Why would a bigger vocabulary reduce token count but increase model size?
• Advanced: Why might the same sentence cost more tokens in Hindi than in English?
• Product-Company: You are billing customers per API call, not per token. Why is that a risky pricing model, and how would you fix it?

Activity#

Guess how a BPE tokenizer might split "internationalization", then run it through a real tokenizer (for example with the code above) and compare. Explain any difference by thinking about which pieces would be frequent in training text.

Homework#

Take one paragraph and its translation in a non-Latin script. Count tokens for both with your chosen model’s tokenizer. Compute the ratio and write two sentences on the cost implication for a global product. Use the rough “4 characters per token for English” rule only as a sanity check and label it as an estimate.

Quick Revision#

• Tokenization = text to subword tokens (BPE) to IDs to learned embedding vectors.
• Vocabularies differ per model family and change over time; recent ones are often 100k or more (some above 200k), and some older ones are about 50k.
• Vocab size trades tokens-per-text against embedding-table size.
• Non-English text, code and numbers often cost more tokens.
• Count with the tokenizer of the model you will call.

Check yourself#

• Walk-through: Why does the same sentence cost different token counts across model families, and what follows for cost and context budgeting?
• Variation: A user pastes 20 pages of Hindi text. Why might cost and latency surprise you, and how would you budget?
• Common misconception: Treating a token as a word, or as a fixed 4 characters for every language.

Related#

• How LLMs work
• Embeddings and similarity search