An LLM can return JSON that is not the JSON your code expects. Structured output means the model produces text in a defined shape, such as a JSON object that matches a JSON Schema definition or a Pydantic model. The model can return malformed JSON or JSON that parses but is missing required fields. It can also return an object that matches the schema but has wrong values, and it may refuse the request or stop early when it runs out of tokens. If your application parses that output and passes it into business logic, each of those failures becomes a wrong order, a bad database write, or an unintended tool call.
The safe path is a four-stage pipeline. Constrain the model at generation time when the provider supports it, then validate every response against your schema before the application trusts it. Handle refusals and truncation as explicit control-flow branches, and bound retries and repairs with a fail-closed fallback. A provider’s JSON mode guarantees only that the output is valid JSON, not that it matches your schema. Even a provider that enforces the shape does not guarantee that the values are correct. You validate at the application boundary anyway.
Why raw model output is not a contract
Models fail structured output in predictable ways. The output may not parse because of missing commas, unclosed brackets, or extra text after the JSON ends. It may parse but miss required fields or use the wrong types. It can also match the schema and still be wrong: a model asked to extract findings from a medical report can return an empty findings array even though the source text clearly contains findings. The object passes validation, but the result is useless. Refusals and incomplete outputs may not fit the expected shape at all, so your code needs branches for those states, not just a parser.
Tool calls are structured output your code then executes. The model proposes a function name and an arguments object, and a wrong tool name or a missing argument can trigger side effects your code never intended. If you treat the tool call as a trusted contract, a model mistake reaches production fastest. The validation rules in the next sections apply to tool arguments with the same force they apply to any other structured response.
Constrain generation, then validate the result again
You get the strongest protection by stopping the model from producing invalid JSON in the first place. Constrained decoding turns a JSON Schema into a finite state machine and blocks tokens that would produce output outside the schema. OpenAI’s Structured Outputs feature, launched in August 2024, uses this approach. The launch evaluation reported that gpt-4o-2024-08-06 scored 100% on complex JSON schema evals with Structured Outputs enabled, while gpt-4-0613 scored under 40%.
JSON mode is the older, weaker option. It guarantees the output parses as JSON, not that it matches a schema, which is why OpenAI recommends Structured Outputs over JSON mode when possible. Constrained decoding also does not guarantee value correctness, only structure. The model can still return a schema-valid object with a hallucinated ID or an empty list where the answer belongs.
Providers differ in which JSON Schema features they accept. OpenAI supports a subset of JSON Schema for Structured Outputs. It requires the root to be an object and every field to be required, and it documents limits on nesting depth and total property count. Some Anthropic paths do not accept constraint keywords such as minimum or maximum, so a Pydantic field with ge/le can produce a schema the provider rejects. Keep the schema small: a five-field intent object that describes what the model decided is more reliable than a 60-field backend object. Use small enums, make required fields explicit, and keep nesting shallow.
Even with all that, validate the result again. A provider or gateway can return a schema-conforming object, and your application still needs to parse it with Pydantic or Zod and run database lookups. It also needs policy checks and postconditions before the object is safe to trust.
Validate the parsed object before trusting it
Parse the object first, then run the semantic checks. In Python, Pydantic’s model_validate_json is usually better than calling json.loads first and then validating, because you do extra parsing work with the two-step route. If you are not using Pydantic, the jsonschema library validates a parsed instance against a JSON Schema directly, and check-jsonschema wraps it as a CLI and pre-commit hook.
After the object parses, run the checks that a schema cannot express. Cross-field rules such as "if the summary says no issues, the issues array must be empty" belong in a Pydantic model_validator that runs after the whole object is parsed. Semantic checks look for empty result lists that should not be empty, IDs that do not exist, and counts that do not match. They also flag citations that do not point to a real document and sections the model never covered. For criteria that are hard to express programmatically, Instructor’s llm_validator makes a second, cheaper model call to check them.
Handle refusals, truncation, and bad tool calls as their own branches
A refusal or a truncated response is a separate state from a schema failure, and your code must branch on it. In the OpenAI API, check response.status == "incomplete" and content.type == "refusal" explicitly. If the stop reason is max_tokens or the finish reason is length, the missing closing brace is not the real problem. The model ran out of room. Retry with a higher token limit or a smaller input.
Tool calls need the same branch. Validate the tool name against an explicit allowlist and validate the arguments dict with the same Pydantic model you use in tests before execution. If validation fails, reject the call and report the error back to the agent instead of executing a partially valid argument dict.
Retry, repair, and fall back with bounds
When validation fails, you have two options: send the error back to the model and ask it to try again, or repair the output locally. Both need bounds because each retry is a full model call.
Send the failed output and the validation error back to the model in a repair prompt, and ask for a corrected response that satisfies the schema. Instructor’s retry loop does exactly this, capturing validation errors, formatting them as feedback, and asking the model to try again. Cap the attempts, log the final failure, and surface a controlled error to the caller.
Heuristic repair with json_repair fixes common mistakes such as missing commas and unclosed brackets without another model call. The tool fills gaps with nulls and empty strings, which can hide a truncated answer, so use it with caution and re-validate after repair. A small model reformer can also fix broken JSON between steps; in one evaluation, a fine-tuned Mistral-7B with constrained decoding reached 99.5% schema accuracy and 94% content similarity. Because repair can change content, verify that the repair did not alter values before you accept the output.
Prefer deterministic correction when the defect has a deterministic cause. If you keep seeing the same defect, such as a missing closing brace caused by truncation, fix the token limit instead of patching the output.
When every attempt fails, return a well-defined fallback value rather than propagating null or crashing, and log the failure. Fail closed instead of normalizing garbage into something that only looks acceptable. For high-stakes governance decisions, fail-closed routing refuses any result that resolves to a different provider and writes an immutable audit record of every denial. The default gateway posture is fail open and can silently substitute a different model.
Plan for partial failures in multi-step pipelines
In an agent pipeline, each LLM call is a step that can fail, and failures compound. With a 5% per-step schema failure rate, a 12-step run has roughly a 46% chance that at least one step produces invalid output.
Do not validate every delta as it arrives; buffer until the API reports a finished message, concatenate the text, then parse once. Libraries such as Instructor and the Vercel AI SDK support partial streaming validation that parses partial JSON as it arrives and yields type-safe partial objects, so you can render early results while still checking the final object.
When a step returns a partially valid object, treat the missing or invalid fields as a failure and retry the whole object with the validation error as feedback. Do not silently fill gaps with nulls or empty strings; that produces an object that looks acceptable but carries less information than the model intended. If the failed step is a tool call, your application must not execute a partially validated argument dict. Reject the call, report the error, and let the agent recover with a corrected call.
