Getting an LLM to return structured data your code can parse is one of the most common production needs. In 2026 there are three distinct mechanisms for this: JSON mode, function calling (tool use), and strict structured outputs. They look similar but have very different reliability guarantees. This article walks through what each one does, how they differ in guaranteed correctness, and which to pick for which use case.
What each mechanism actually does
JSON mode (OpenAI, Anthropic, Google): the model is nudged by the system prompt and inference-time logic to return valid JSON. The output will parse as JSON, but the shape of the JSON is not guaranteed. You asked for {name, email}; the model might return {full_name, email_address} on a bad day.
Function calling / tool use (OpenAI tools, Anthropic tools, Google function calling): you define a JSON Schema for a function. The model returns a tool call whose arguments conform to the schema. Reliability depends on the model; in 2026, frontier models hit the schema correctly 95%+ of the time but can still drift on edge cases.
Strict structured outputs (OpenAI’s response_format with strict mode, Anthropic’s structured output as of 2026): token-by-token constrained decoding guarantees the output matches a JSON Schema. The model literally cannot emit a token that would violate the schema. 100% reliability.
OpenAI strict mode (the gold standard)
Added mid-2024, matured through 2026. Pass a response_format with a Pydantic model or raw JSON Schema and strict: true:
from openai import OpenAI
from pydantic import BaseModel
class Ticket(BaseModel):
category: str
priority: str
needs_followup: bool
summary: str
client = OpenAI()
completion = client.beta.chat.completions.parse(
model="gpt-4o-2024-08-06",
messages=[
{"role": "system", "content": "Classify this support ticket."},
{"role": "user", "content": ticket_text},
],
response_format=Ticket,
)
ticket = completion.choices[0].message.parsed # Already a Ticket instanceThe parse helper validates the response against the Pydantic model. If the model refuses (safety) you get message.refusal instead of parsed. There is no “did it return the right shape?” question anymore because the shape is enforced at the decoder level.
Function calling for tool-using agents
Function calling fills a different role: it lets the model choose from multiple possible actions, each with its own schema. The model can call one function, multiple functions, or no function at all. Use this when the output is “which tool and with what args”, not “parse this document”.
tools = [
{
"type": "function",
"function": {
"name": "search_products",
"description": "Search our catalog",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string"},
"max_price": {"type": "number"},
},
"required": ["query"],
"additionalProperties": False,
},
"strict": True,
},
},
{
"type": "function",
"function": {
"name": "check_order_status",
"description": "Look up order status",
"parameters": {
"type": "object",
"properties": {"order_id": {"type": "string"}},
"required": ["order_id"],
"additionalProperties": False,
},
"strict": True,
},
},
]
response = client.chat.completions.create(
model="gpt-4o",
messages=messages,
tools=tools,
)Add strict: true to each function and the arguments are now schema-constrained like structured outputs. Combine function calling (choose which tool) with strict mode (guarantee arg shape) for the most reliable agent setup.
Anthropic’s approach
Claude supports tool use with JSON Schema arguments. Claude Sonnet 5.5 and Opus 5.5 hit the schema correctly in production without strict constrained decoding, though Anthropic added strict schema enforcement for tools in 2026:
from anthropic import Anthropic
client = Anthropic()
response = client.messages.create(
model="claude-sonnet-5-5",
max_tokens=1024,
tools=[{
"name": "classify_ticket",
"description": "Classify a support ticket",
"input_schema": {
"type": "object",
"properties": {
"category": {"type": "string", "enum": ["bug", "feature", "billing"]},
"priority": {"type": "string", "enum": ["low", "medium", "high"]},
},
"required": ["category", "priority"],
},
}],
tool_choice={"type": "tool", "name": "classify_ticket"},
messages=[{"role": "user", "content": ticket_text}],
)
tool_use = next(b for b in response.content if b.type == "tool_use")
result = tool_use.input # dict matching the schemaSetting tool_choice to a specific tool forces the model to call it (not optional). This is Anthropic’s equivalent of “give me structured output in this shape”.
Google Gemini structured outputs
Gemini supports both response_schema (for direct JSON output) and function calling. The response_schema path is cleanest:
from google import genai
from pydantic import BaseModel
class Ticket(BaseModel):
category: str
priority: str
client = genai.Client()
response = client.models.generate_content(
model="gemini-2.5-pro",
contents="Classify: My card was charged twice",
config={"response_mime_type": "application/json", "response_schema": Ticket},
)
ticket = response.parsed # Pydantic instanceReliability is high but not 100% constrained-decoding like OpenAI strict mode. For critical paths, add a Pydantic validation step after parse.
When to pick which
- Document parsing or extraction: strict structured outputs (OpenAI
response_formatwithstrict: true). Guaranteed shape, no retries. - Agents choosing between tools: function calling with strict mode enabled on each tool. Model picks the right tool and gets the args shape right.
- Legacy / cheap models: JSON mode as the fallback with Pydantic validation and 1 retry on validation failure.
- Mixed natural + structured output: function calling where the function captures the structured bit and the model’s unstructured response is the natural part.
Common schema gotchas
- OpenAI strict mode schema restrictions: no
oneOf, limitedanyOf, every property must be inrequired,additionalPropertiesmust befalse. Convert flexible schemas into strict-compliant shapes. - Enum unknowns: if a field has 20 valid values and the model sees an edge case, strict mode forces it into one of the 20 (loss of information). Add an “other” enum value if real-world data has long tail.
- Deeply nested schemas: 5+ levels deep tends to confuse models and hurt accuracy. Flatten where you can.
- Numbers vs strings: if the data is “sometimes a number, sometimes a string” (phone numbers, IDs), pick string everywhere. Multi-type fields degrade reliability.
- Date/time formats: specify format in the description. The model is less likely to guess if you say “ISO 8601 like 2026-10-15T14:30:00Z”.
Latency and cost tradeoffs
Strict constrained decoding has marginal latency overhead (the decoder evaluates grammar constraints on every token). In practice this adds 5% to 15% latency vs free-form generation. Token cost is identical: schemas do not count toward input tokens separately.
JSON mode is slightly faster than strict mode (no grammar enforcement) but requires retry logic in production. The retry cost outweighs the per-call latency savings; strict mode is almost always the right pick when available.
Frequently Asked Questions
Official documentation
Can I use strict mode with any OpenAI model?
Only models trained with strict support: gpt-4o-2024-08-06 and later, gpt-4o-mini (2024-07-18 and later), o1, o3. Older models like gpt-4-turbo do not support strict mode and will error.
What happens if the schema is impossible to satisfy?
The model emits whatever it can within the schema and may truncate at the output token limit. Set a reasonable max_tokens; if hit, extend it. The output will still parse as the correct shape but values may be partial or placeholder.
Does strict mode cost more?
No. Token cost is identical to free-form generation on the same model. The grammar enforcement happens inside the inference server, not as additional model calls.
Can I use Pydantic validators inside a strict schema?
Pydantic validators run after the model has produced the output. Strict mode guarantees type/shape but not business logic (an email string being a syntactically valid email, a value in a valid range). Add Pydantic validators for business rules on top of strict-mode shape guarantees.
Should I use JSON mode or strict mode for new code?
Strict mode, every time, if the model supports it. JSON mode is legacy at this point; the only reason to use it is older models that do not support strict output.
How do I handle refusals in strict mode?
Check message.refusal on the parsed response. If non-null, the model declined to answer (safety reasons). Treat refusals as a separate failure path from schema failures; log them separately in observability so you can tell “model refused” apart from “no match” in your analytics.
Adrian Mercurio
Full-Stack Developer at PIES IT Solution
Specializes in building complete capstone projects with full documentation. Strong background in PHP/MySQL development and database design. Has personally built and tested over 30 capstone-ready projects with ER diagrams, DFDs, and chapter-by-chapter thesis documentation.
Expertise: PHP · Laravel · Database Design · Capstone Projects · C# · C · C++ · Python · AI Projects · View all posts by Adrian Mercurio →