Lesson deck
Tokenization and embeddings flashcards
Reviewing needs JavaScript. You can read every card in the list below.
Loading your cards...
Back to the lesson6 cards in this deck
Read all 6 cards as a list
- Front
- What is BPE and why is it used?
- Back
- Byte-Pair Encoding repeatedly merges the most frequent adjacent pair of bytes/characters to build a subword vocabulary. Any text, even an unseen word, can then be written from a fixed set of pieces.
- Front
- Why do LLMs struggle with spelling and digit-level arithmetic?
- Back
- They see subword tokens, not characters or place-value digits. Tool use (code or a calculator) is the robust fix.
- Front
- How big are tokenizer vocabularies today, and is it the same for every model?
- Back
- No. Each model family trains its own. Public examples: tiktoken gpt2 has 50,257; cl100k_base is about 100k; o200k_base about 200k; Llama 3 uses 128K (sources checked 2026-10-09). Check the model you use.
- Front
- What is the vocabulary-size trade-off?
- Back
- A bigger vocabulary gives fewer tokens per text (cheaper, shorter sequences) but a larger embedding table and output layer.
- Front
- Are token embeddings fixed vectors?
- Back
- No. The embedding table has one row per token and is learned jointly with the rest of the model.
- Front
- How should you count tokens for a cost estimate?
- Back
- Use the tokenizer or token-counting tool for the exact model you will call, and measure real traffic in each language. Another family's count, or a '4 characters per token' rule, is only an estimate.