Getting Reliable Structured Outputs from LLMs: Validation, Retries and Constrained Decoding

The Problem of Premature Trust

The common industry pitfall is to treat a successful JSON parse as the final step of a pipeline. However, a model can generate perfectly valid JSON that is semantically catastrophic. For instance, consider an automated invoice processing system tasked with extracting an invoice_id, currency, total_cents, and a total_quote. A model might output a valid JSON object where the total_cents field contains a calculation error—perhaps 12,500 cents for an invoice clearly stating a total of 1,250.00 EUR. Because the schema itself is satisfied—the output is a valid integer—the application layer may blindly commit this data to a database.

This disconnect between structural validity and factual accuracy necessitates a multi-layered verification architecture. Relying solely on a schema is insufficient; the output must be validated against the source material through a rigorous, multi-stage "contract" check.

Defining the Contract: The First Line of Defense

Modern data validation requires a contract that the application, not the model, dictates. Utilizing tools like Pydantic, developers can define strict schemas that forbid extra keys and disable default type coercion. By setting strict=True and extra="forbid", the application ensures that the model cannot "hallucinate" extraneous data or silently convert incompatible types, such as treating a string "125000" as a valid integer.

In a professional production environment, this contract serves as a gatekeeper. If the model returns data that deviates from the expected schema, the ValidationError generated by Pydantic provides specific, actionable feedback. This feedback is not just for logging; it is essential for the second stage of the pipeline: intelligent retries.

Chronology of an Extraction Pipeline

A resilient extraction pipeline follows a strict chronological order of operations:

1. Request: The model is queried with a prompt, a source document, and a schema constraint.
2. Structural Validation: The response is parsed. If the JSON is malformed or violates the Pydantic schema, a shape_error is triggered.
3. Semantic Verification: If the shape is valid, the data is compared against the source text. This includes verifying that the total_quote actually appears in the source and that the total_cents calculation is mathematically consistent with the text.
4. Outcome Routing: The pipeline resolves to one of three states: accepted (data is valid), needs_review (the model correctly identified missing data as null), or rejected (the model produced a factual error or the attempt budget was exceeded).

The Logic of Bounded Retries

A common mistake in early AI integration is the "infinite loop," where the system continues to re-query the model until it receives a valid-looking answer. This is not only inefficient but also dangerous. A sophisticated pipeline should only retry on a shape_error. If the model produces a syntactically correct object that is semantically wrong—such as a miscalculated total—the pipeline should reject the response immediately.

If the invoice itself contains an error, or if the model simply cannot extract the data, a retry will not resolve the issue. By limiting attempts to a maximum of three, developers can prevent unnecessary costs and avoid the potential for the model to "guess" to satisfy the loop.

Constrained Decoding: Reducing the Failure Surface

Constrained decoding is the most significant leap forward in LLM reliability. By masking out tokens that would lead to invalid JSON during the generation process, the model is physically incapable of emitting a malformed structure. This is accomplished via libraries that integrate with the model’s tokenizer to force compliance with a grammar.

When using high-level SDKs like client.responses.parse, the developer delegates the structural enforcement to the model provider. However, even with constrained decoding, the semantic check remains essential. A model can be constrained to produce a perfect integer for total_cents but still produce the wrong integer. The architecture must remain decoupled: the decoder ensures the shape, but the application ensures the truth.

Broader Implications and Strategic Analysis

The shift toward structured output reflects a broader maturation of the AI industry. We are seeing a move away from "black box" prompt engineering toward deterministic software engineering. The implications are profound for industries such as finance, legal services, and healthcare, where data integrity is non-negotiable.

Data from recent industry benchmarks suggest that when developers implement rigorous validation loops and constrained decoding, error rates in structured extraction tasks drop from nearly 15% to under 1%. The remaining 1% of errors are typically "semantic anomalies"—cases where the document format is ambiguous or the source text is inherently contradictory. In these instances, the pipeline must be designed to hand off the task to a human agent, marking the status as needs_review.

Strategic Recommendations for Implementation

1. Ownership of the Schema: The schema must be defined in the application code, not as part of the prompt. This ensures that the code can evolve without requiring a prompt rewrite.
2. The "Human-in-the-Loop" Threshold: Never allow the system to default to an empty or zeroed-out value if the model cannot find the information. A null value, explicitly returned, should trigger a human review.
3. Transport vs. Content: Distinguish between network-level retries (e.g., HTTP 429 errors) and content-level retries (e.g., shape errors). These should be handled by separate mechanisms to avoid masking underlying logic issues with transport-layer fixes.
4. Auditability: Every step of the process—from the initial raw text to the final validated object—should be logged with clear identifiers. This allows for post-mortem analysis of why a specific extraction failed, which is vital for continuous improvement of the model’s system prompt.

As we look toward 2027 and beyond, the expectation is that model providers will continue to integrate these validation layers directly into their infrastructure. However, the responsibility for verifying the accuracy of the extracted content against the source of truth will remain firmly with the application developers. By treating LLM outputs as untrusted input that must pass through a multi-gate validation process, developers can finally harness the power of generative AI without compromising the integrity of their business-critical data.