Cheat sheet: LLM foundations

LLM inference loop: tokenize → embed + position → transformer blocks (causal self-attention + feedforward) → logits → softmax → sample → append → repeat until stop token or length limit.

Self-attention: Q/K/V per token; score = softmax(QKᵀ/√d), with a causal mask so a token sees only itself and earlier tokens; output = weighted sum of V. Parallel across positions in training and prefill, so it beats RNNs on parallelization and long-range links. The score matrix is n × n: attention-score compute is quadratic in sequence length during prefill (total cost grows less steeply at short lengths because other layers are linear).

Prefill vs decode: prefill processes the whole prompt in parallel (sets time to first token). Decode makes one token per step and is typically memory-bandwidth-bound.

KV-cache: reuse past tokens’ K/V; compute K/V only for the new token each step. Saves compute, costs memory: 2 × layers × KV-heads × head-dim × sequence length × batch × bytes per value. It is not a response cache.

Tokenization: BPE/SentencePiece give subword units, not words. Each model family has its own tokenizer and vocabulary; sizes vary and change (public examples checked 2026-10-09: tiktoken gpt2 50,257, cl100k_base about 100k, o200k_base about 200k; Llama 3 128K). Non-English text, code and numbers often cost more tokens. Token count ≠ word count, and it drives cost and context budget. Count with the model’s own tokenizer.

Sampling:

• Temperature divides the logits: higher is flatter and more random; lower is sharper. At 0 it is greedy, but not guaranteed deterministic (inference kernels are not batch-invariant, so batching and load can change results).
• Top-k keeps the k highest tokens. Top-p keeps the smallest set whose cumulative probability reaches p, so it adapts to the distribution.
• Low temperature for extraction and classification; higher for creative tasks. Change one control at a time. Some newer models do not accept sampling parameters; check the docs.

Open-weight vs API-only: open-weight means downloadable weights (self-host, fine-tune, data stays with you; you run the serving and ops; read the license). API-only means no infrastructure and often frontier quality, but less control and your data goes to the provider. Decide with your own evaluation set, cost, latency and data constraints.

Interview traps: “it just predicts words” (no mechanism) · assuming a bigger context is free or replaces retrieval · assuming 1 token = 1 word · assuming temperature 0 is deterministic.

Quick answers#

• LLM step: logits → softmax → sample → append → repeat (autoregressive).
• Attention: softmax(QKᵀ/√d)V; parallel; quadratic score matrix; causal mask; needs positional information.
• KV-cache: reuse past K/V in decode; memory grows with layers × KV-heads × sequence × batch.
• Sampling: temperature reshapes logits; top-p adapts the cutoff; T=0 does not guarantee identical outputs.
• Tokens: subwords; cost and context are counted in tokens; tokenizers differ by model.

Traps#

“Predicts words” (no mechanism) · KV-cache = response cache · assuming 4 chars/token everywhere · tuning T and top-p together.