Generative AI
SearchStart here

Lesson deck

How LLMs work flashcards

Reviewing needs JavaScript. You can read every card in the list below.

Loading your cards...

Back to the lesson8 cards in this deck

Read all 8 cards as a list
  1. Front
    What does an LLM actually output at each generation step?
    Back
    A probability distribution over its vocabulary for the next token. One token is sampled, appended to the sequence, and the process repeats.
  2. Front
    Why is self-attention parallelizable in a way RNNs aren't?
    Back
    Every token's Query/Key/Value can be computed and compared with the other visible tokens in a few large matrix operations during training. An RNN needs the previous step's hidden state before it can do the next step.
  3. Front
    How does self-attention cost grow with sequence length n, and what does the causal mask do?
    Back
    The score matrix is n x n, so the score arithmetic is quadratic in n (prefill). The causal mask blocks each token from attending to later tokens so the model cannot peek at the answer.
  4. Front
    What problem does KV-caching solve?
    Back
    It avoids recomputing Key/Value vectors for all previous tokens at every decode step. Only the new token's K/V is computed and appended.
  5. Front
    Why divide attention scores by the square root of d_k?
    Back
    Dot products grow with vector size. Scaling keeps their variance near 1 so softmax does not saturate into a near one-hot output, which would starve the training signal.
  6. Front
    Prefill vs decode: which is typically compute-bound and which memory-bound?
    Back
    Prefill processes the whole prompt in parallel and keeps the GPU busy. Decode generates one token per step and is typically limited by memory bandwidth (reading weights and the KV-cache).
  7. Front
    Write the KV-cache size formula. What does it limit in production?
    Back
    2 x layers x KV-heads x head-dim x sequence length x batch x bytes per value. It grows linearly with context and with concurrent requests, so it often caps how many requests one GPU can serve. GQA/MQA shrink it by using fewer KV-heads.
  8. Front
    Why can't attention alone distinguish word order?
    Back
    Without position information, attention is permutation-equivariant: shuffle the input tokens and the outputs are shuffled the same way, with no sense of order. Positional information (learned, sinusoidal or rotary) must be injected.