Generative AI
SearchStart here

Module deck

LLM foundations flashcards

Reviewing needs JavaScript. You can read every card in the list below.

Loading your cards...

Back to the module20 cards in this deck

Read all 20 cards as a list

How LLMs work

  1. Front
    What does an LLM actually output at each generation step?
    Back
    A probability distribution over its vocabulary for the next token. One token is sampled, appended to the sequence, and the process repeats.
  2. Front
    Why is self-attention parallelizable in a way RNNs aren't?
    Back
    Every token's Query/Key/Value can be computed and compared with the other visible tokens in a few large matrix operations during training. An RNN needs the previous step's hidden state before it can do the next step.
  3. Front
    How does self-attention cost grow with sequence length n, and what does the causal mask do?
    Back
    The score matrix is n x n, so the score arithmetic is quadratic in n (prefill). The causal mask blocks each token from attending to later tokens so the model cannot peek at the answer.
  4. Front
    What problem does KV-caching solve?
    Back
    It avoids recomputing Key/Value vectors for all previous tokens at every decode step. Only the new token's K/V is computed and appended.
  5. Front
    Why divide attention scores by the square root of d_k?
    Back
    Dot products grow with vector size. Scaling keeps their variance near 1 so softmax does not saturate into a near one-hot output, which would starve the training signal.
  6. Front
    Prefill vs decode: which is typically compute-bound and which memory-bound?
    Back
    Prefill processes the whole prompt in parallel and keeps the GPU busy. Decode generates one token per step and is typically limited by memory bandwidth (reading weights and the KV-cache).
  7. Front
    Write the KV-cache size formula. What does it limit in production?
    Back
    2 x layers x KV-heads x head-dim x sequence length x batch x bytes per value. It grows linearly with context and with concurrent requests, so it often caps how many requests one GPU can serve. GQA/MQA shrink it by using fewer KV-heads.
  8. Front
    Why can't attention alone distinguish word order?
    Back
    Without position information, attention is permutation-equivariant: shuffle the input tokens and the outputs are shuffled the same way, with no sense of order. Positional information (learned, sinusoidal or rotary) must be injected.

Tokenization and embeddings

  1. Front
    What is BPE and why is it used?
    Back
    Byte-Pair Encoding repeatedly merges the most frequent adjacent pair of bytes/characters to build a subword vocabulary. Any text, even an unseen word, can then be written from a fixed set of pieces.
  2. Front
    Why do LLMs struggle with spelling and digit-level arithmetic?
    Back
    They see subword tokens, not characters or place-value digits. Tool use (code or a calculator) is the robust fix.
  3. Front
    How big are tokenizer vocabularies today, and is it the same for every model?
    Back
    No. Each model family trains its own. Public examples: tiktoken gpt2 has 50,257; cl100k_base is about 100k; o200k_base about 200k; Llama 3 uses 128K (sources checked 2026-10-09). Check the model you use.
  4. Front
    What is the vocabulary-size trade-off?
    Back
    A bigger vocabulary gives fewer tokens per text (cheaper, shorter sequences) but a larger embedding table and output layer.
  5. Front
    Are token embeddings fixed vectors?
    Back
    No. The embedding table has one row per token and is learned jointly with the rest of the model.
  6. Front
    How should you count tokens for a cost estimate?
    Back
    Use the tokenizer or token-counting tool for the exact model you will call, and measure real traffic in each language. Another family's count, or a '4 characters per token' rule, is only an estimate.

Sampling parameters and the model landscape

  1. Front
    What does temperature do, and what happens as it goes to 0?
    Back
    It divides the logits before softmax: higher gives a flatter distribution (more random), lower a sharper one. At 0 the choice is greedy, always the top token, but that does not guarantee identical outputs across runs.
  2. Front
    Top-k vs top-p: what is the difference and when does it matter?
    Back
    Top-k keeps a fixed number of highest-probability tokens. Top-p keeps the smallest set whose cumulative probability reaches p, so it keeps few tokens when the model is confident and more when it is unsure.
  3. Front
    Open-weight vs API-only model: one key trade-off?
    Back
    Open-weight: you can self-host, fine-tune and keep data in your infrastructure, but you own serving and operations. API-only: no infrastructure and often frontier quality, but data goes to the provider and you cannot change the weights.
  4. Front
    Does temperature 0 guarantee identical outputs? Why not?
    Back
    No. A key cause is that inference kernels are not batch-invariant: the result for your request can depend on how many other requests share the batch, and server load changes that (Thinking Machines Lab, 2025-09-10). Store outputs, validate against a schema and do not rely on regeneration.
  5. Front
    Why is a bigger context window not a free upgrade?
    Back
    You pay per token, prefill attention work and KV-cache memory grow with length, and quality can drop for information in the middle of long inputs. It also does not replace retrieval.
  6. Front
    Which sampling settings for extraction vs creative writing, and what caveat?
    Back
    Extraction and classification: low temperature (0 or near it). Creative writing: higher temperature or top-p below 1. Change one control at a time, and check whether your model accepts these parameters at all.