Lesson deck
Sampling parameters and the model landscape flashcards
Reviewing needs JavaScript. You can read every card in the list below.
Loading your cards...
Back to the lesson6 cards in this deck
Read all 6 cards as a list
- Front
- What does temperature do, and what happens as it goes to 0?
- Back
- It divides the logits before softmax: higher gives a flatter distribution (more random), lower a sharper one. At 0 the choice is greedy, always the top token, but that does not guarantee identical outputs across runs.
- Front
- Top-k vs top-p: what is the difference and when does it matter?
- Back
- Top-k keeps a fixed number of highest-probability tokens. Top-p keeps the smallest set whose cumulative probability reaches p, so it keeps few tokens when the model is confident and more when it is unsure.
- Front
- Open-weight vs API-only model: one key trade-off?
- Back
- Open-weight: you can self-host, fine-tune and keep data in your infrastructure, but you own serving and operations. API-only: no infrastructure and often frontier quality, but data goes to the provider and you cannot change the weights.
- Front
- Does temperature 0 guarantee identical outputs? Why not?
- Back
- No. A key cause is that inference kernels are not batch-invariant: the result for your request can depend on how many other requests share the batch, and server load changes that (Thinking Machines Lab, 2025-09-10). Store outputs, validate against a schema and do not rely on regeneration.
- Front
- Why is a bigger context window not a free upgrade?
- Back
- You pay per token, prefill attention work and KV-cache memory grow with length, and quality can drop for information in the middle of long inputs. It also does not replace retrieval.
- Front
- Which sampling settings for extraction vs creative writing, and what caveat?
- Back
- Extraction and classification: low temperature (0 or near it). Creative writing: higher temperature or top-p below 1. Change one control at a time, and check whether your model accepts these parameters at all.