Integrating OpenAI Structured Outputs for Guaranteed JSON Parsing

Extracting data from documents, classifying requests, and building RAG pipelines all share the same headache: invalid JSON output. The model may forget to close a bracket, mix up a field type, or add an extra key. In production with 10,000+ requests per day, the fraction of invalid responses can reach 15–20%. Each such response triggers a retry, burning extra tokens and time. When parsing 50,000 invoices daily, retries cost approximately $2,500 per month. OpenAI Structured Outputs solves this at generation time: constrained decoding guarantees every token respects a given JSON schema. We've implemented this approach in several large projects—here are the technical details, including Pydantic model configuration and batch processing.

Problems We Solve

Without Structured Outputs, developers spend up to 40% of their time on retries and response validation. Typical scenarios:

• Extracting invoice details: the model returns total_amount: "12 345.67" (string) while the schema expects a float, or forgets the vat_amount field.
• Ticket classification: instead of priority: "high" it produces priority: "High" (wrong case)—doesn't match the enum.
• Batch processing: 1000 documents, 20% invalid responses—manual correction.

Structured Outputs eliminates these problems: the response always conforms to the schema. We use it for invoice parsing, ticket classification, and automatic CRM population.

Why Structured Outputs Is More Than Just json_object?

The standard response_format: json_object only asks the model to output JSON but does not enforce a schema. Structured Outputs uses constrained decoding—at each generation step, only tokens that lead to valid JSON per the specified schema are allowed. This yields a 99.5%+ first-attempt success rate in our projects, compared to 80–85% with json_object.

How Pydantic Helps in Python Projects

from openai import OpenAI
from pydantic import BaseModel
from typing import Literal, Optional

client = OpenAI()

class Invoice(BaseModel):
    vendor_name: str
    invoice_number: str
    date: str
    total_amount: float
    currency: str
    line_items: list["InvoiceItem"]
    vat_amount: Optional[float] = None

class InvoiceItem(BaseModel):
    description: str
    quantity: float
    unit_price: float
    total: float

Invoice.model_rebuild()

def extract_invoice(text: str) -> Invoice:
    response = client.beta.chat.completions.parse(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Extract invoice data from the text"},
            {"role": "user", "content": text}
        ],
        response_format=Invoice,
    )
    return response.choices[0].message.parsed

Pydantic automatically validates types on the client, but with Structured Outputs this is redundant: the model already returned correct data. Nevertheless, we keep validation to log mismatches (e.g., unparseable dates).

How to Implement for Batch Processing

In one project we processed 50,000+ documents per day. We used:

• OpenAI GPT-4o (primary model)
• gpt-4o-mini for pre‑classification (cheaper, p99 latency < 2s)
• Pydantic v2 + LangChain for prompt management
• ChromaDB for semantic search over extracted data

Key finding: Structured Outputs reduced API calls by 25% due to zero retries. In batch processing we achieved 99.7% correct extractions on the first attempt.

Classification with Enum

from enum import Enum

class TicketCategory(str, Enum):
    technical = "technical"
    billing = "billing"
    feature_request = "feature_request"
    complaint = "complaint"
    general = "general"

class TicketClassification(BaseModel):
    category: TicketCategory
    priority: Literal["low", "medium", "high", "critical"]
    sentiment: Literal["positive", "neutral", "negative", "angry"]
    requires_human: bool
    summary: str
    tags: list[str]

def classify_ticket(text: str) -> TicketClassification:
    response = client.beta.chat.completions.parse(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"Classify this ticket: {text}"}],
        response_format=TicketClassification,
        temperature=0,
    )
    return response.choices[0].message.parsed

Structured Outputs via JSON Schema (Without Pydantic)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Product data"}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "product_data",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "price": {"type": "number"},
                    "in_stock": {"type": "boolean"},
                    "categories": {
                        "type": "array",
                        "items": {"type": "string"}
                    }
                },
                "required": ["name", "price", "in_stock", "categories"],
                "additionalProperties": False,
            }
        }
    }
)
import json
data = json.loads(response.choices[0].message.content)

What Limitations Must Be Considered?

• strict: True requires additionalProperties: False at all levels.
• Nullable fields via "type": ["string", "null"] are not supported—use anyOf.
• Maximum nesting depth: 5 levels.
• For recursive schemas, use $ref.

When to Choose Structured Outputs vs json_object?

Comparison of the two approaches: Structured Outputs wins 3–5× in schema accuracy on test samples, but adds ~15% to generation time. For realtime scenarios (chatbots), use gpt-4o-mini—it's fast and cheap.

Latency and Accuracy Comparison

Process and Timelines

1. Analysis: measure current parsing errors, identify document types (invoices, waybills, tickets).
2. Schema design: create Pydantic models, test on sample data.
3. Implementation: integrate Structured Outputs, handle edge cases, add logging.
4. Testing: A/B test comparing extraction quality before/after.
5. Deployment: containerization, monitor latency and valid response rate.

Estimated timelines:

• Basic integration with one schema: 1–2 days.
• Complex pipeline with multiple document types and RAG: 1–2 weeks.

Get a free engineer consultation—discuss your use case. Request Structured Outputs integration.

Common Implementation Mistakes

• Omitting additionalProperties: False—the model adds extra keys.
• Using temperature > 0.3—increases risk of deviating from the schema.
• Ignoring logging—without monitoring, rare failures go undetected.
• Attempting to parse nested objects deeper than 5 levels—API limitation.

These mistakes reduce the rate of correct responses by 10–30%. Our engineers know how to avoid them. Contact us for a project assessment.

LLM Development: Fine-Tuning, RAG, Agents, and Production Deployment

Using GPT‑4 or Claude 3.5 Sonnet through a public API is not a solution — it's just a tool. When the requirement is to "make it like ChatGPT, but on our data," there is a real engineering challenge behind it: from prompt engineering to training a 70B model on your own infrastructure. End-to-end LLM solution development is a complex stack, and we have been doing it for over 5 years. During this time, we have completed over 20 projects in generative AI: from RAG systems for legal departments to custom support agents. Where exactly your task falls depends on data, latency requirements, budget, and how critical confidentiality is.

A typical situation: the client has already tried ChatGPT, but results are unstable — sometimes accurate, sometimes hallucinating. Or they need integration into a corporate portal while complying with security policies. Let's break down each layer of the stack in detail — from RAG to production deployment.

Why Do RAG Systems Break and How to Fix It?

RAG (Retrieval-Augmented Generation) looks simple: find relevant documents, put them in context, get an answer. In practice, it fails in several places.

Chunking without overlap. Classic mistake: chunk_size=512, overlap=0. If the answer lies across two chunks, retrieval won't find either with sufficient confidence. Solution: overlap 15–25% of chunk_size, or better yet, sentence-aware splitting with spaCy or NLTK instead of naive character splitting.

Poor embedder. text-embedding-ada-002 is good for general use, but on legal or medical texts, specialized models like E5-large-v2, BGE-M3, or fine-tuned sentence-transformers on domain data outperform it. Recall@5 differences can be 15–25%.

No re-ranking. Vector search optimizes for speed, not relevance. A cross-encoder re-ranker (ms-marco-MiniLM-L-6-v2, bge-reranker-large) after initial retrieval improves top-3 accuracy with acceptable latency (+50–150ms). This is often more impactful than improving the embedding model.

Hybrid search. Dense vectors alone work poorly on exact queries: names, SKUs, codes. BM25 (sparse) finds exact matches but misses semantics. Hybrid via RRF (Reciprocal Rank Fusion) is the optimal compromise. Qdrant, Weaviate, and pgvector 0.7+ support hybrid search natively.

When to Fine-Tune Instead of Prompt Engineering?

Prompt engineering solves ~70% of LLM adaptation tasks for a domain. The remaining 30% require fine-tuning. Three indicators: the model ignores a specific output format even with detailed prompting; the task requires deep knowledge of specialized vocabulary (medicine, law); you need to significantly reduce token costs by replacing a large model with a smaller specialized one.

LoRA and QLoRA are the standard for SFT. LoRA adds trainable low-rank matrices to attention layers. A typical configuration for Llama-3 8B: r=64, lora_alpha=128, target_modules=["q_proj","v_proj","k_proj","o_proj"] yields ~0.8% trainable parameters, training on one A100 40GB. QLoRA adds 4-bit quantization (NF4) and allows fine-tuning 70B models on two A100 40GB, though speed drops by half compared to bf16.

DPO instead of RLHF. Direct Preference Optimization requires only (chosen, rejected) pairs, not scalar reward signals. DPOTrainer from the trl library (Hugging Face) implements it in a few dozen lines.

Common mistake. A dataset of 500 examples, 5 epochs, validation loss 0.8 — seems fine. But on test, the model degrades on general instructions. Cause: catastrophic forgetting. Solution: add 10–20% general instruction-following examples (Alpaca, FLAN) to the training set to preserve original capabilities.

How to Choose a Base Model: 8B or 70B?

For most tasks, fine-tuning an 8B model is sufficient. 70B is needed when deep reasoning is required or the 8B baseline does not reach the required quality even after fine-tuning. Inference cost for Llama-3 8B via vLLM on A100 is efficient; the exact cost depends on volume.

What Does PagedAttention Bring to Production?

vLLM is the first choice for serving open-source models. PagedAttention is the key technical innovation: KV-cache is managed like virtual memory in an OS, without fragmentation. This yields 2–4x higher throughput compared to naive HuggingFace Transformers inference. The vLLM documentation confirms that continuous batching and PagedAttention are the standard for high-load LLM services.

Typical numbers on A100 80GB for Llama-3 8B (bf16): 400–600 req/s, P50 latency 200–400ms, P99 latency 600–900ms at concurrency 64. For 70B on two A100 with tensor parallelism: 80–120 req/s, P99 latency 1.5–2.5s. AWQ or GPTQ quantization reduces memory consumption by 2x with quality loss within 1–3%.

Multi-Agent Systems

Agents are LLMs with access to tools: search, code execution, API calls, database interaction. Common patterns:

• ReAct (Reason + Act): the model reasons → chooses a tool → observes the result → reasons again. LangChain and LlamaIndex implement it out of the box.
• Multi-agent orchestration: multiple specialized agents with a coordinator on top. Example: coordinator → researcher (search + summarization) → coder (code generation and execution) → critic (verification). Tools: AutoGen (Microsoft), CrewAI, custom implementation on LangGraph.

In production, agent systems are non-deterministic. Essential: guardrails, step limits, logging of each step, human-in-the-loop for critical actions.

How We Work: Stages, Timeline, Deliverables

What Is Included

We deliver:

• Technical documentation (model card, configs, deployment instructions)
• Access to infrastructure (code repository, trained weights)
• 1 month of post-deployment support (consultations, bug fixes)
• Customer team training (2–3 sessions on system operation)

Timeline: basic RAG prototype — 1–2 weeks. Fine-tuning with customer data — 3–6 weeks (including data preparation). Production system with monitoring and retraining — 2–4 months. Cost is calculated individually based on data volume, model complexity, and infrastructure requirements.

We guarantee the quality of the final model with performance benchmarks and ongoing monitoring. Our engineers have hands‑on experience with dozens of production LLM systems.

Want to evaluate your project? Leave a request — we will prepare a preliminary summary within 1–2 business days. Or get a consultation on choosing the approach: RAG, fine-tuning, or hybrid — we will tell you what works best for you. Contact us to discuss your LLM development needs. Schedule a free consultation today.