Reliably getting structured outputs from LLMs in production

Structured output is one of the oldest unsolved problems in applied LLM work. You ask the model for JSON. It gives you JSON — most of the time. Occasionally it gives you JSON with a trailing comment. Sometimes it gives you JSON wrapped in a markdown code fence. Every few thousand requests it gives you a confident English paragraph explaining what the JSON would contain if it were going to produce it, which it has decided not to.

In a prototype, this is annoying. In a production pipeline, it is a reliability incident waiting to happen. The parser throws an exception, the job fails, an alert fires, someone wakes up.

This post covers the full stack of techniques for making structured LLM output reliable: what each provider’s JSON/structured mode actually guarantees, how to validate and recover from failures, retry strategies that don’t compound costs, and where gateway-layer output validation fits in.

What providers actually guarantee

“JSON mode” and “structured outputs” are distinct features across providers, and the guarantees differ meaningfully.

JSON mode (available on OpenAI, Anthropic, and most major providers) constrains the model to produce only valid JSON tokens. It does not constrain the schema. You get syntactically valid JSON, but the fields may not match what you asked for, values may be wrong types, and required fields may be missing.

Structured outputs (OpenAI’s name for their schema-constrained mode, introduced in 2024) uses constrained decoding — the token sampler is filtered at inference time to only allow tokens that keep the output on a valid path through your JSON schema. This provides a much stronger guarantee: the output will be valid JSON that conforms to your schema. Fields will be present, types will match, enum values will be valid.

Tool use / function calling is the oldest approach: you define a function with a typed parameter schema, ask the model to call it, and parse the arguments. This works reliably across providers, but the interface is more verbose to construct and varies more between providers.

The hierarchy of reliability, roughly: constrained decoding > tool use > JSON mode > unconstrained with system prompt instructions.

For new production code, prefer constrained decoding or tool use whenever the provider supports it. Do not rely on system prompt instructions alone — “always respond in JSON” is not a constraint, it is a request.

Defining a schema worth validating

A common mistake is defining a schema that is too permissive. If every field is type: string and nothing is required, the schema does not actually express your intent, and validation against it will pass even when the output is semantically wrong.

Define your schemas with the precision you would use for a database table:

{
  "type": "object",
  "required": ["category", "confidence", "extracted_entities"],
  "properties": {
    "category": {
      "type": "string",
      "enum": ["billing", "technical", "account", "general"]
    },
    "confidence": {
      "type": "number",
      "minimum": 0.0,
      "maximum": 1.0
    },
    "extracted_entities": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["text", "type"],
        "properties": {
          "text": {"type": "string"},
          "type": {"type": "string", "enum": ["product", "person", "date", "amount"]}
        }
      }
    },
    "requires_escalation": {
      "type": "boolean"
    }
  },
  "additionalProperties": false
}

The enum constraint on category is load-bearing. Without it, the model might produce "billing_issue" or "Billing" when your downstream system expects exactly "billing". The additionalProperties: false prevents the model from adding helpful but unexpected fields that your parser might not handle.

Validating the response before trusting it

Even with constrained decoding, validate the parsed output against your schema in application code. The defense-in-depth principle applies: provider guarantees can have edge cases, providers can update their constrained decoding implementation, and your schema might have gaps you did not anticipate.

Use a proper JSON Schema validation library, not manual field checks:

import jsonschema
import json

def validate_llm_response(raw: str, schema: dict) -> dict:
    try:
        parsed = json.loads(raw)
    except json.JSONDecodeError as e:
        raise StructuredOutputError(f"Invalid JSON: {e}", recoverable=True)

    try:
        jsonschema.validate(parsed, schema)
    except jsonschema.ValidationError as e:
        raise StructuredOutputError(
            f"Schema validation failed: {e.message}",
            path=list(e.absolute_path),
            recoverable=True
        )

    return parsed

The recoverable=True flag is significant. A JSON parse error or schema validation failure is a signal to retry with a clarifying prompt. A business logic error in the validated output (a confidence score of 1.0 on something that should be uncertain) is a different kind of problem that retrying will not fix.

Retry strategies

When validation fails, you have three options: fail the request, retry with the same prompt, or retry with a corrective prompt.

Fail immediately is appropriate for hard budget constraints or when the downstream system can handle null responses gracefully. It is usually wrong for user-facing features.

Retry with the same prompt is the simplest approach and works when the failure was caused by model stochasticity — the model occasionally produces malformed output that it would get right 95% of the time. One retry catches most of these. Two retries catches almost all. Beyond two retries you are probably fighting a systematic issue, not stochasticity.

Retry with a corrective prompt adds the failed response and the validation error back into the conversation:

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": user_prompt},
    {"role": "assistant", "content": failed_response},
    {
        "role": "user",
        "content": (
            f"Your previous response failed schema validation: {validation_error}. "
            f"Please respond again with valid JSON matching the required schema. "
            f"Do not include any explanation — only the JSON object."
        )
    }
]

This technique works well for complex schemas where the model misunderstood an enum value or omitted a required field. It costs an extra round-trip but often recovers without escalating to a fallback model.

Implement retry with exponential backoff and jitter, and cap total retries at 2–3. Log every retry attempt with the validation error — this data is invaluable for diagnosing whether a specific schema or prompt is causing systematic failures.

Fallback model routing

If your primary model fails to produce valid structured output after retries, routing to a different model is a valid recovery strategy. Some models are significantly better than others at following complex schema constraints, and the performance gap varies by schema type.

For structured output tasks in particular, it is worth benchmarking models against your specific schemas during evaluation, not just on generic benchmarks. A model that scores well on general reasoning may struggle with your specific enum constraints or nested object requirements. The right primary model for structured output may not be the same as the right model for open-ended generation.

A fallback chain for a classification task might look like:

route:
  primary: openai/gpt-4o-mini
  fallbacks:
    - model: openai/gpt-4o
      trigger: structured_output_validation_failure
      max_attempts: 1
    - model: anthropic/claude-3-5-haiku
      trigger: structured_output_validation_failure
      max_attempts: 1

The fallback to a larger model on validation failure is a deliberate cost trade-off: you pay more for the retry request but maintain reliability for the end user.

Gateway-layer output validation

Application-layer validation catches schema failures after the response has been returned from the gateway to your application. For many architectures this is fine. But there are cases where you want validation to happen at the gateway layer, before the response is handed to the application:

• Shared infrastructure: if multiple applications use the same gateway and each one would need to implement its own validation, centralizing it at the gateway reduces duplication and ensures consistent behavior.
• Audit and observability: you want a single place to log structured output failure rates, retry counts, and model-level performance on schema compliance — not scattered across application logs.
• Policy enforcement: certain outputs may need to be blocked or transformed regardless of which application receives them. A guardrail that detects when a model is about to return a confidence score of 1.0 on a high-stakes classification task can intercept that and request a retry before the application ever sees it.

A gateway-layer output validation policy attaches a JSON schema to a route. Any response that fails schema validation triggers the configured recovery action — retry, fallback, or error — before the response is returned downstream.

Observability for structured output pipelines

The metrics you need to operate a structured output pipeline reliably:

Schema validation failure rate — the percentage of responses that fail validation before any retry. This is your primary signal for whether your prompt or schema needs tuning. A rate above 1–2% warrants investigation.

Retry success rate — of the requests that initially failed, what fraction succeeded on retry? If retry success rate is low, you have a systematic problem that retries are not fixing.

Model-level failure rate — break down validation failures by model. If one model in your fallback chain has a consistently higher failure rate on your schema, route around it or remove it from the chain.

Latency impact of retries — retries add latency. Track the p99 latency of requests that required at least one retry, and ensure your application’s timeout budget accounts for the retry overhead.

Cost per successful structured output — total tokens spent (including retry tokens) divided by successful responses. This is the actual cost of structured output reliability at your validation failure rate.

Without these metrics, you are guessing at whether your reliability measures are working. Log every validation attempt, every retry, and every fallback invocation as structured events.

Schema design patterns that reduce failures

A few schema design choices consistently reduce validation failure rates:

Avoid deep nesting: schemas with objects inside arrays inside objects are harder for models to follow than flat schemas. If you can denormalize, do it.

Use explicit null over optional fields: instead of making a field optional (which some models interpret as “I can omit it”), make it required but nullable. The model then has to actively decide to emit null rather than forgetting the field.

Add descriptions to enum values: constrained decoding handles enum enforcement, but adding descriptions helps the model choose the right value, not just a syntactically valid one.

Bound numeric ranges: minimum and maximum on confidence scores and other bounded values prevent the model from producing semantically invalid outputs that pass structural validation.

Keep arrays bounded: if an array should never have more than 10 items, say so with maxItems. An unbounded array is an invitation to over-generate.

ManyLayers Gateway supports route-level structured output validation policies — schema attachment, retry configuration, and fallback chains — with per-model failure rate metrics surfaced in the analytics dashboard. If you are managing this stack of logic across multiple application teams independently, centralizing it at the gateway layer is worth examining.