Getting a language model to return a JSON object occasionally is easy. Getting it to return valid, schema-consistent JSON reliably across thousands of calls, with diverse inputs, under adversarial edge cases, is a different problem. Most engineers underestimate the gap between the two.
This is a practical guide to the techniques that actually close that gap — not the ones that work in demos but collapse in production.
The most common approach to structured output is to append an instruction like "return your response as JSON" to the system prompt and assume the model will comply. This works often enough in development to seem reliable. In production, it breaks on inputs the developer did not test: topics that make the model verbose, edge cases where the model hedges with prose before the JSON, content that the model wraps in markdown code fences, outputs where a field value contains unescaped quotes.
The instruction "return JSON" tells the model the format but not the schema. The model has to infer the schema from context, and that inference is not stable across all inputs. What you need is an explicit, documented schema that the model can treat as a contract.
The most robust technique for structured output is to put the full schema definition directly in the system prompt, describe each field and its type, and provide multiple concrete examples of valid and invalid outputs. This sounds like a lot of prompt real estate, and it is. The tradeoff is reliability.
For JSON arrays of typed objects — which is the common pattern for document generation — the schema definition should specify the exhaustive list of permitted type values, the required fields for each type, and what constitutes a schema violation. The model should be explicitly told that any deviation from the schema is a pipeline failure, not a partial success. That framing shifts the model toward compliance rather than creative interpretation.
Examples in the prompt should include at least one correct example for each object type and one clearly labeled incorrect example with an explanation of why it fails. The incorrect examples teach the model the failure modes it should avoid, which is often more effective than the correct examples alone.
Language models are trained on code and documentation where JSON is typically wrapped in markdown code fences (triple backticks with a json language tag). This training creates a strong prior toward wrapping JSON output in fences even when explicitly instructed not to. The instruction "no markdown formatting" reduces but does not eliminate this behavior.
The practical fix is to strip markdown fences in post-processing rather than relying entirely on the model to omit them. This is a one-time implementation cost that removes an entire class of parse failures. The stripping logic should handle the case where the fence includes a language tag (```json), the case where it does not (```), and the case where there is prose before or after the fenced block.
Even with a well-designed prompt and fence stripping, parse failures will happen. The question is what to do when they do. Failing the entire pipeline on a JSON parse error is too brittle — the model may have produced a document that is 95% correct with a single unescaped character breaking the parse.
The correction pass pattern is to detect a parse failure, send the broken output back to the model with the parse error and an explicit instruction to fix only the syntax errors, and attempt to parse again. This catches the majority of failures because they are typically caused by a small number of syntax issues rather than fundamental structural problems. The correction pass should be limited to one retry to prevent the pipeline from spinning indefinitely on unfixable output. This is one of several trust boundary patterns discussed in Building Multi-Agent Pipelines That Actually Work.
Fields with a constrained set of valid values — type fields, status fields, category labels — are a common source of subtle schema violations. The model may return a value that is semantically equivalent to a valid enum but not lexically identical: "type_" instead of "type", "warning_icon" instead of "warning", a color as "red" instead of a hex code.
The prompt should enumerate the valid values explicitly and state that any other value is invalid. It should also give examples of common near-misses that are not valid, because the model needs to see the failure mode to reliably avoid it. At the schema validation layer, reject near-misses rather than coercing them — silent coercion hides the fact that the model is drifting from the schema and makes it harder to know when prompt changes are necessary.
Structured output prompts degrade as the model changes and as input distributions shift. A prompt that achieves high parse reliability on the inputs present at development time may perform worse on new topic domains or after a model update. The only way to know is to measure.
A minimal test harness for structured output should track parse success rate, schema validation pass rate, and the distribution of failure types across a representative sample of inputs. Run this against new model versions before switching to them. Treat a parse success rate below 95% as a prompt engineering problem, not a model limitation — in almost all cases, the prompt can be improved to close the gap.