Reasoning models and structured outputs

Story

Hide story

The dispatch clerk asks Tobias which carts are going where. Tobias gives a lovely answer about a busy evening with several departures, and some lovely weather for it.

The clerk does not want a speech. The ledger machine takes one line per cart, with the same four fields every time, and it has just rejected the whole paragraph.

What does it take to ask for an answer a machine can read, and what happens on the nights it still comes back wrong?

Concept#

Reasoning modelsReasoning modelA model trained to spend extra output on working through a problem before answering. It helps hard multi-step tasks and costs more time and tokens.Open the glossary entryClose think in a long chain of intermediate text before they give the final answer, and they spend more inference-time compute (also called test-time computeTest-time computeExtra computation spent when answering, rather than in training, to get better results. Longer reasoning, sampling several answers and verifying are examples.Open the glossary entryClose) on harder problems. It is chain-of-thought built into the model: instead of you asking for step-by-step reasoning in the prompt, the model does it by default or when you switch it on.

Structured outputsStructured outputA reply that follows a defined shape, such as JSON matching a schema. When the provider enforces the schema, the form is reliable but the values can still be wrong, and a truncated reply or a refusal may not match, so validate anyway.Open the glossary entryClose make the model’s reply machine-readable, so downstream code can rely on its shape.

Both are trade-offs between quality, latency, cost and reliability, and both are common practical interview topics.

Why It Matters#

Two decisions come up in nearly every production LLM feature. First: which model and setting handle this request? A reasoning model can be much better on hard problems and wasteful on easy ones, and you pay for the thinking whether or not it helped. Second: how do you get output a program can parse? A parse error in production is an outage for the feature, and “the model usually returns valid JSON” is not a reliability story.

Mental Model#

A standard model answers from instinct. A reasoning model works on scratch paper first. Scratch paper helps on math, code, planning and multi-step logic. It is slow and costs money on questions where the answer is obvious.

For structured output, think of a form with fields instead of a blank page. A form can guarantee that every field is filled in the right type. It cannot guarantee that what you wrote in the field is true.

Technical Deep Dive#

What a reasoning model does and what it costs#

Reasoning tokensReasoning tokensTokens a reasoning model generates while thinking. They are billed as output and count toward the output limit even when the API does not return them.Open the glossary entryClose are generated before the visible answer and are billed as output tokens, even when the API does not show them to you (Anthropic thinking docs; OpenAI reasoning guide, both checked 2026-10-09). They also count toward the output limit, so a small max_tokens can leave no room for the answer. Both providers expose how many tokens were spent on reasoning in the usage data, which is what you watch to see what a setting really costs.

Providers also give you a control for how hard the model thinks, for example an effort setting in Anthropic’s API and a reasoning.effort setting in OpenAI’s. Names, allowed values, defaults and which models accept them change often. On recent Claude models effort is described as a behavioural signal, not a strict token budget, and the older fixed budget_tokens setting is deprecated or rejected on newer models (Anthropic effort and extended thinking docs, checked 2026-10-09). The durable idea: lower effort means faster and cheaper with some loss of capability on hard tasks, so set it from measurements, and read the docs for the model you actually call.

Three honest limits:

• Latency and tokens grow. Thinking happens before the first visible token of the answer.
• Overthinking and diminishing returns. More thinking does not always mean a better answer, and easy tasks gain little.
• The visible reasoning is not a guaranteed record. In Anthropic’s hint experiment (contrived multiple-choice tests on two early-2025 reasoning models), the models mentioned a hint they used in only 25% (Claude 3.7 Sonnet) and 39% (DeepSeek R1) of cases on average (Anthropic, 2025). Do not treat reasoning text as an audit trail.

Other ways to spend inference compute#

These work with any model, including non-reasoning ones:

• Chain-of-thought prompting (Wei et al., 2022).
• Self-consistency: sample N reasoning paths and take the majority answer (Wang et al., 2022). Cost is about N times one call.
• Best-of-N with a verifier: generate N answers and keep the one a checker or reward model scores highest.
• Tool-assisted verification: run the code, check against tests, query the database.

Prompting a reasoning model#

Give the goal, the constraints and the output format. Do not script every step. OpenAI’s guidance for its reasoning models is a clear goal, strong constraints and an explicit output contract “without prescribing every intermediate step” (OpenAI reasoning guide, checked 2026-10-09). Treat the effort setting as a tuning knob and measure; it is not the first way to fix a weak prompt.

When to use a reasoning model#

The table is a starting hypothesis, not a rule. The way to decide is to run both on a labelled set of your own tasks and compare the quality gain with the extra latency and cost. A reasoning model that is 8 percentage points better but triples p95 latency may be right for a nightly report and wrong for a chat reply. Routing by task (easy requests to a fast model, hard ones to a reasoning model) often beats one choice for everything.

Structured outputs: the options#

These are different tools with different guarantees. They are not a single ladder from bad to good.

Sources: OpenAI structured outputs guide (JSON mode gives valid JSON; only Structured Outputs enforces your schema; OpenAI recommends structured outputs over JSON mode) and Anthropic structured outputs docs (constrained decodingConstrained decodingRestricting which tokens the model may produce at each step so the output must fit a schema. The shape conforms; the content can still be wrong.Open the glossary entryClose from a compiled grammar; strict tool use validates tool names and inputs), both checked 2026-10-09.

So how do you choose? Use schema-constrained structured outputs when you want the model’s reply to the user or to your pipeline to have a fixed shape (extraction, classification, a report object). Use tool calling when the model needs to act or fetch something through your functions. With strict tool use, the arguments are schema-checked too, so the two overlap, but the purpose differs: one shapes a reply, the other triggers an action. Prompt-and-parse is acceptable for a prototype only.

What constrained decoding still does not protect you from#

Even with a schema guarantee, check these:

• Truncation. If the output hits the token limit, it can be incomplete and not match the schema. Both providers document this; check the stop reason or status.
• Refusals. A refusal can come back without following your schema. Check for it before parsing.
• Unsupported schema features. Providers support a subset of JSON Schema. For example, Anthropic documents that recursive schemas and numeric range constraints such as minimum and maximum are not supported in strict mode, so range checks stay in your own validation code. Unsupported features return a 400 error, so you find out loudly, not silently.
• Enum casing. Anthropic notes it does not guarantee the capitalisation of enum values, so compare in a case-insensitive way.
• First-request latency. A new schema must be compiled, so the first call with it is slower. Anthropic caches compiled grammars for 24 hours from last use, so later calls are not.
• Truth. A well-formed {"total": 99999} can still be wrong.

Design advice that follows: keep schemas simple, use descriptive field names and enums instead of free text, and leave a field for notes or reasoning, placed before the answer fields, so a strict schema does not squeeze out the model’s working. The field-order advice is common practice, not provider guidance, so test it on your own cases.

Validate, retry, fall back#

Treat model output like input from an untrusted user.

1. Parse and validate against the schema with Pydantic or JSON Schema.
2. Check semantics the schema cannot: ranges, totals that add up, dates that exist, an ID that exists in your database.
3. Retry with the error. On failure, send the validation error back and ask for a corrected output. Allow a small, fixed number of retries (two is common), because every retry costs money and time.
4. Fall back. After the limit, do something defined: route to a human queue, use a default, or return a clear error. Never pass unvalidated output on.
5. Log the failure type and rate, so you can see when a model or prompt change breaks things.

Diagram#

Diagram: Each task is routed by difficulty, then by latency budget (fast model with chain-of-thought if it is tight) and cost budget (fast model if...

Real-World Application#

A support-triage pipeline (illustrative design, not a measured result): a fast model with schema-constrained output classifies the ordinary tickets. Only ambiguous, high-value tickets go to a reasoning model. Every output is validated before it touches the CRM, and anything that fails twice goes to a person. The share of tickets that need the expensive path is a number you measure on your own traffic, not one you assume.

Interview Perspective#

Expect: “what is a reasoning model and when would you not use one”, “how do you guarantee valid JSON”, and “what does self-consistency cost”. Common mistakes: using a reasoning model everywhere, believing constrained decoding makes the content correct, and having no retry or fallback plan. A strong answer to “guarantee valid JSON” names the schema-constrained option, then lists what it does not cover (truncation, refusals, truth) and the validate-retry-fallback loop.

Provenance: Common industry pattern. Standard topics for GenAI roles, not a verified list for any named company.

Practice Questions#

• Basic: What is the difference between JSON mode and schema-constrained structured outputs?
• Intermediate: Why can self-consistency improve accuracy, and what does it cost?
• Advanced: A reasoning model improved accuracy by 8 percentage points but p95 latency tripled. How do you decide?
• Product-Company: Design an invoice extraction pipeline with a 99.9% parse-success target. What do you measure to know whether you reached it?

Activity#

Take six tasks (classify a ticket, extract fields from an invoice, summarise a thread, plan a database migration, debug a failing test, solve a logic puzzle). For each, pick a fast or a reasoning model with a one-line reason and the eval you would run to check the choice. Then pick one task and write down what a wrong answer would cost, to see whether the extra latency is justified.

Homework#

Write a Pydantic or JSON Schema model for an invoice (vendor, date, line items, total). Write a validate-and-retry loop in pseudocode or Python that includes a semantic check (line items sum to the total), a retry limit, a fallback, and the metrics you would log. Say which structured-output option you would use and which failure cases it still leaves to you.

Quick Revision#

• Reasoning models spend extra inference compute: better on hard multi-step tasks, slower and costlier, reasoning tokens billed as output.
• The visible reasoning is a clue, not a guaranteed record.
• Decide fast versus reasoning by measuring on your own tasks, and route by task.
• JSON mode gives valid JSON. Schema-constrained outputs give schema-conformant JSON. Tool calling is for actions.
• Constrained decoding guarantees form, not truth. Still handle truncation and refusals, validate semantics, retry a bounded number of times, and have a fallback.

Check yourself#

• Walk-through: Your team wants to move everything to a reasoning model. What do you say, and what do you measure?
• Variation: Make LLM output safe to feed a payments API.
• Common misconception: “Schema-valid means correct”, and “tool calling is the strongest form of structured output.”

Related#

• Prompting fundamentals
• Function and tool calling
• Cost and latency optimization