Lensa ML
Lensa ML

Structured Outputs

Step 1 of 6: The Problem

Free sampling can break the format

Temperature T0.80
Samples N100
Sample 1, prompt: "Return the user Ada, age 36, as JSON"{"name": "Ada", "age": 36} <eos>All 100 samples, one square eachWhy the invalid ones failText before the JSON14Age not a number7Missing field5Stopped early2Syntax error7Toy model, illustrative logits over a 26-token vocabulary
seed 1
Valid outputs
65 / 100
Failure rate
35.0%
Most common failure
Text before the JSON

Sample 1 parses, but 35 of the 100 samples do not. Raising T spreads probability onto the tokens that break the format, and the failure rate climbs.

Drag Temperature T and Samples N, then press Resample to draw a new set of replies. A language model has no built-in notion of JSON. It learned the format from text, so every token it emits is a probability, and over many calls the rare bad tokens show up. Even one stray character makes json.loads raise an error. A prompt that says "reply only with JSON" lowers the failure rate but never brings it to zero, and the cost grows with the number of calls a pipeline makes.

Step 2 of 6: Logit Masking

Illegal tokens get a logit of −∞

Temperature T1.00
Output so far (token 1 of 11, state s1){"softmax without maskafter the maskmasked p{"95.7%Sure−∞```−∞{4.3%":␣−∞thirty−∞,␣"−∞",−∞":−∞}−∞Top 10 of 26 toy tokens by logit (illustrative). Red rows break the schema.
Allowed tokens
2 / 26
Mass removed
30.1%
Picked token
{"
Its p after mask
95.7%

Here 30.1% of the unmasked probability sits on tokens that break the schema, led by Sure. Setting their logits to −∞ gives that mass to the 2 legal tokens.

Press Play, or use Step and Back to move one token at a time, and drag T to see how the unmasked bars spread. Masking happens between the model's forward pass and the sampler: logits come out, illegal entries become −∞, and softmax gives them exactly zero probability. The model weights never change. Because the remaining probabilities are renormalized, temperature, top-k, and top-p still work on the allowed set. Greedy decoding under the mask picks the highest legal logit at every step.

Step 3 of 6: Grammar to State Machine

A schema compiles to an FSM

Characters read0
Schema {"name": string, "age": int}, read one character at a time{"name": "Ada", "age": 36}{"name":␣"a-za-z,␣"age":␣0-90-9end24 states. Each box shows the character it accepts next; loops repeat a class.
Current state
s1 of 24
Allowed next
{
Accepting
no
Toy tokens allowed
2 / 26

Inside fixed text the FSM has one way forward: the next character must be "{". Any other character, or a token that contains one, is rejected.

Drag Characters read across the example object and follow the highlighted state. A regular expression compiles to a finite-state machine, and a JSON schema with fixed keys and simple types can first be written as such a regex. This toy layout fixes the whitespace; real compilers allow optional spaces and newlines, which adds branches. Nested objects and arrays of any depth need a stack, so libraries handle them with a context-free grammar and a pushdown automaton instead of a plain FSM.

Step 4 of 6: Token Boundaries

Precompute allowed tokens per state

FSM state (s1 to s24)6
Shortest text that reaches s7, then the index row for s7{"name▏{rejected{"rejected"→ s8namerejectedagerejected":→ s9 (2)":␣"→ s11 (4)":␣→ s10 (3)",rejected,␣"rejected␣"rejected␣rejected:rejectedAdarejectedArejecteddarejectedthirtyrejected36rejected3rejected6rejected}rejected36}rejected"}rejectedSurerejected```rejected<eos>rejectedBlue (n): the token covers n grammar symbols. Built once, before decoding.
Allowed here
4 / 26
Span 2+ symbols
3
Toy index entries
624
With a 32,000-token vocab
768,000

The single token ":␣" covers 4 grammar symbols and jumps from s7 straight to s11. Matching whole tokens to whole symbols would reject it, so the index walks each of its characters through the FSM.

Drag FSM state from s1 to s24 and compare which of the 26 chips light up. Willard and Louf (2023) proposed building this index once per schema: for every state, walk each vocabulary token through the machine and store the tokens that survive along with their end states. Decoding then needs one lookup per step instead of a scan of the vocabulary. With the 7B chat model's 32,000 tokens and these 24 states, the index takes 768,000 token walks, done before the first request.

Step 5 of 6: Cost and Quality

Field order changes the answer

Temperature T1.00
Prompt: "What is 17 × 24? Reply as JSON."{"answer": ▮, "reasoning": "..."}softmax without maskint mask appliedmasked p40840.4%41824.5%39814.8%41012.2%3888.1%Let−∞I−∞"−∞Illustrative logits for the first token of the answer value. Correct: 408.
p("408") after mask
40.4%
Mass removed by mask
58.2%
Mask work vs forward pass
1 : 421,250

The model wants to start explaining ("Let", "I"), but the int field forces a number now. The mask moves 58.2% of the probability onto digits, and "408" ends up with 40.4% against 24.5% for "418".

Switch between Answer first and Reasoning first, then drag T. The mask itself is cheap: applying it touches 32,000 logits, while one forward pass of a 7B model costs about 2 × 6.74 billion, or 13.5 billion, FLOPs. The quality cost is different. Tam et al. (2024) found that strict format rules can lower scores on reasoning tasks. Because the model writes fields in schema order, a reasoning field placed before the answer acts as a short chain of thought inside the JSON.

Step 6 of 6: In Practice

JSON mode, schemas, and tools

Calls allowed (retry until valid)1
Prompt onlyJSON modeConstrainedValid JSONFits the schemaFails per callTypical toolsnot alwaysnot always43.5%alwaysnot always13.9%alwaysalways0.0%instructions, then parse and retryPrompt only: chance of a valid reply within k calls56.5%k = 181.1%k = 291.8%k = 396.4%k = 498.4%k = 5Failure rates from the step 1 toy model at T = 0.8, 1,000 samples
Fails per call
43.5%
Valid within 1 call
56.5%
Expected calls per valid reply
1.77

With only instructions, 43.5% of replies fail to parse or to match the schema. 1 call brings the success rate to 56.5%, at 1.77 calls per valid reply on average.

Pick Prompt only, JSON mode, or Constrained, then drag Calls allowed to see how retries add up. Function and tool calling uses the same machinery: the tool's argument schema becomes the constraint for that part of the reply. Outlines implements the FSM index, XGrammar handles context-free grammars, and llama.cpp accepts GBNF grammar files. Serving engines such as vLLM expose guided decoding through these backends. Several hosted APIs also accept a JSON schema directly, so the constraint runs on the provider's side.

What are structured outputs in LLMs?

Structured outputs are model replies that are guaranteed to follow a fixed format, such as JSON that matches a given schema. The usual method is constrained decoding: at each step, any token that would break the format gets its logit set to −∞ before sampling. The schema compiles to a state machine, and the machine decides which tokens are legal next.

Programs that call a language model usually need data, not prose. Extracting fields from a document, filling a form, classifying a ticket, and calling a tool or function all expect a reply that a parser can read on the first try. Constrained decoding removes the parse-and-retry loop, keeps field types correct, and lets small local models produce output that downstream code can trust.

Key ideas

  • Sampling freely from a language model gives valid JSON most of the time, but never every time.
  • Constrained decoding sets the logit of every format-breaking token to −∞, so those tokens get zero probability.
  • A regex or simple JSON schema compiles to a finite-state machine that lists the legal next characters in each state.
  • Tokens often span several grammar symbols, so the allowed tokens per state are found by walking each token through the machine, once, ahead of time.
  • The mask costs almost nothing per step, but the field order in a schema can change answer quality.
  • JSON mode guarantees valid syntax only; schema-constrained decoding also guarantees the fields and types.

Common questions

Does constrained decoding guarantee correct answers?

No. It guarantees the shape of the output, not the truth of its values. A field typed as an integer will hold an integer, but it can still be the wrong integer. Validation of the content, such as range checks or cross-checks against a source, is still needed.

What is the difference between JSON mode and structured outputs?

JSON mode constrains the reply to be syntactically valid JSON, with any keys and value types. Structured outputs constrain it to one specific schema: the required keys, their types, and allowed enum values. Code that uses JSON mode still has to check the schema itself.

Does constrained decoding slow down generation?

Per token the overhead is small once the index or grammar is compiled, since the mask is a lookup plus an operation over the vocabulary. Compiling a large or complex schema can take noticeable time on the first request, which is why libraries cache compiled grammars.

Can constrained decoding work with any model?

It works with any model that exposes its next-token logits, because it only changes which tokens may be sampled. That includes local open-weight models. With a hosted model, it depends on whether the provider runs the constraint on its side.

Related explainers

References

  1. Willard and Louf (2023), "Efficient Guided Generation for Large Language Models"
  2. Dong et al. (2024), "XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models"
  3. Tam et al. (2024), "Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models"

Last updated