Sampling parameters and the model landscape

Story

Hide story

Three times in ten minutes a passenger asks Tobias which platform the 22:10 leaves from. He answers warmly each time: Platform 2, then Platform 4, then Platform 2 again.

Marta Voss, passing with an empty thermos, does not ask which answer was right. “Is he broken,” she says, “or is that how a Voice works? And do we keep this one, or hire another?”

Where would you start?

Concept#

After the model computes logits (How LLMs work), it does not have to pick the single most likely token. How you choose is called decoding, and a few parameters control it:

• Greedy decodingGreedy decodingAlways picking the single most likely token. It is what temperature 0 amounts to, and it still does not guarantee identical outputs run to run.Open the glossary entryClose always takes the highest-probability token.
• TemperatureTemperatureA sampling setting that divides the logits before softmax. Higher values flatten the distribution (more random); lower values sharpen it. At 0 the choice is greedy, which is more repeatable but not guaranteed identical across runs.Open the glossary entryClose divides the logits before softmax. Higher gives a flatter distribution and more randomness. Lower gives a sharper distribution. As temperature approaches 0, sampling approaches greedy.
• Top-k keeps only the k highest-probability tokens, then samples among them.
• Top-p (nucleus sampling)Top-p (nucleus sampling)A sampling filter that keeps the smallest set of tokens whose cumulative probability reaches p. It keeps few tokens when the model is confident and more when it is unsure.Open the glossary entryClose keeps the smallest set of tokens whose cumulative probability reaches p, then samples among them (Holtzman et al., 2019).

Why It Matters#

Two production questions come straight from this. Why does the same prompt give different answers? (Sampling with temperature above 0.) And how do I get consistent output when I need it, for example in classification or extraction? Temperature 0 is the first thing most people reach for, and it helps, but it does not give a guarantee. That gap, and what to do about it, is the most useful thing in this lesson.

Mental Model#

Temperature is a confidence dial on the model’s own ranking. Near 0, the model commits to its top choice. Higher up, it will sometimes pick the second or third choice. It does not make the model smarter or more creative on its own. It changes how often a less likely word is chosen.

Top-k and top-p are bouncers at the door. Top-k lets in the best k candidates no matter what. Top-p lets in the best candidates until their combined probability is high enough, so it lets in few when the model is sure and more when the model is unsure.

Example: what temperature does to a distribution#

Illustrative numbers. Three candidate tokens have logits 2.0, 1.0 and 0.0.

Lower temperature piles probability onto the top token. Higher temperature spreads it out.

Now top-p. With probabilities 0.97, 0.02, 0.01 and p = 0.9, top-p keeps only the first token. Top-k with k = 2 would still keep two, including a 2% option. With probabilities 0.40, 0.35, 0.25 and p = 0.9, top-p keeps all three while top-k with k = 2 drops a 25% candidate. That is what “adaptive cutoff” means.

Technical Deep Dive#

Choosing settings#

Two cautions. Do not tune temperature and top-p together by feel, because they both reshape the same distribution and their effects overlap. Change one, measure, then decide. And check your provider: some current models do not let you set these at all. For example, Anthropic’s migration guide says that on Claude Sonnet 5.5 a non-default temperature, top_p or top_k returns a 400 error, while Claude Sonnet 4.6 and earlier models, and Claude Haiku 4.5, accepted them (Anthropic docs, checked 2026-10-09). Treat sampling controls as model-specific, and read the current docs for the model you call.

Temperature 0 is not a determinism guarantee#

In theory, temperature 0 is greedy decoding and therefore repeatable. In practice, hosted inference often returns different text for the same prompt. A common explanation blames floating-point math and parallel threads. Floating-point addition is not associative, so summing the same numbers in a different order can give slightly different results, and that is the root ingredient. But the Thinking Machines Lab analysis (Horace He and colleagues, 2025-09-10) argues that the common explanation is incomplete, because the order of operations in a fixed batch is stable. Their finding: the forward pass kernels are run-to-run deterministic for a fixed batch, but they are not batch-invariantBatch invarianceThe property that a request's result does not depend on which other requests share its batch. Many inference kernels lack it, which is one reason temperature 0 is not fully deterministic.Open the glossary entryClose, so the result for your request depends on how many other requests the server batched with it. Server load changes batch size unpredictably, so batch variance exposes the floating-point effect and your output varies. They showed that with batch-invariant kernels, 1,000 completions of the same prompt came out identical (source).

What this means for you as an engineer:

• Do not promise bit-for-bit repeatability from an API just because temperature is 0. Some providers offer a seed or similar control; treat it as best effort unless the provider documents otherwise.
• For audit or compliance, store the output that was produced and shown instead of expecting to regenerate it later.
• Make downstream steps robust: constrain output with a schema, validate it, and compare with tolerance (for example, the extracted fields, not the exact prose).
• If you need exact repeatability and control the stack, self-hosting with deterministic or batch-invariant kernels is possible but is an engineering and performance decision.

Open-weight versus API-only models#

• Open-weight modelsOpen-weight modelA model whose weights you can download. You can self-host and fine-tune it and keep data in your infrastructure, but you run serving and operations and must read the license.Open the glossary entryClose publish their weights so you can download and run them. For example, Meta released Llama 3 weights (Meta, 2024-04-18). You can self-host, fine-tune and keep data inside your infrastructure, but you pay for GPUs, serving and operations. Licenses vary and “open-weight” is not the same as open source, so read the license.
• API-only (proprietary) models are used through a provider’s API. You manage no infrastructure, and the strongest models are often here, but your data goes to the provider (subject to its policies), you cannot change the weights, and the provider can change or retire the model.

Neither is better by default. Choose on measurable criteria: quality on your own task, cost at your volume, latency, data-residency and compliance, and the operations team you actually have. Then measure: build a small evaluation set from real examples, run both options on it, and compare quality, cost per request and p95 latency.

Context windows are not free#

A large context windowContext windowThe maximum number of tokens a model can take in and produce in one request. A larger window is not free: you pay per token, and attention work and KV-cache memory grow with length.Open the glossary entryClose is an engineering and cost decision.

• You are billed on tokens in and out, often at different rates for input and output.
• Prefill attention work grows with the square of the prompt length, and the KV-cache memory grows linearly with context (see How LLMs work). Providers and kernels optimize this, so measure the cost you actually see instead of assuming a formula.
• Quality can drop: Liu et al. (2023) found that models do best when relevant information is at the start or end of a long input and worse when it is in the middle.
• Big windows do not remove the need for retrieval: retrieval selects what is relevant, controls cost, and lets you update knowledge without retraining. Retrieval-augmented generation is the subject of later modules.

Diagram#

Diagram: Logits are divided by the temperature, then filtered either by keeping the k highest tokens (top-k) or the smallest set whose cumulative...

Real-World Application#

Choose settings per use case: low temperature for extraction and classification, higher for generative tasks, and fixed settings when you evaluate changes. Then design as if outputs can still vary: schema validation, retries and stored outputs. Choosing open-weight or API-only based on data privacy, cost and customization is a recurring system-design discussion, and the answer is a measured comparison on your own data.

Interview Perspective#

Expect: “how do you make LLM output deterministic” and “when would you self-host an open-weight model instead of calling an API”. A strong answer to the first says low temperature helps, then explains why it is not a guarantee and what you do about it. A common mistake is to assume a bigger context window replaces retrieval. It does not: more context is not better recall, and it costs more.

Provenance: Common industry pattern. Standard topics for GenAI roles, not a verified list for any named company.

Practice Questions#

• Basic: What does temperature = 0 do to generation, and what does it not guarantee?
• Intermediate: Why might top-p be preferred over top-k in some cases?
• Advanced: Why is “just use a 1M-token context window instead of RAG” often the wrong answer in an interview?
• Product-Company: You need outputs as consistent as possible, and auditable, for a compliance use case. What sampling settings do you use, and what else do you build around them?

Activity#

Pick two tasks, such as writing a product tagline and extracting a phone number from text. Justify the sampling settings for each. Then run the same extraction prompt several times at your lowest available temperature and note whether the output ever differs. If it never does, explain what your test could not tell you (for example, whether the server was under load).

Homework#

Look up the current context window and the open or closed status of two model families. Write two sentences on which you would choose for a data-sensitive internal tool, and list the three criteria you would measure before committing. Cite the page you used and the date.

Quick Revision#

• Temperature reshapes the distribution; top-k and top-p filter it; top-p adapts to how peaked it is.
• Temperature 0 is greedy in principle and not guaranteed identical in practice. Design for variation.
• Some models do not accept sampling parameters at all; check the docs.
• Open-weight: control and data locality plus operations burden. API-only: convenience and often frontier quality, less control.
• A larger context window costs money and can lower quality; it does not replace retrieval.

Check yourself#

• Walk-through: Pick sampling settings for (a) SQL generation, (b) marketing copy, (c) a classifier. Justify each and say whether temperature 0 is fully deterministic.
• Variation: Choose between a small open-weight model and a frontier API for a 50,000 requests-per-day summarisation task. Give three criteria and a measurement plan.
• Common misconception: Tuning temperature and top-p together blindly; assuming the biggest model or the longest context is always better.

Related#

• How LLMs work
• Cost and latency optimization