AI Test Data Generation: From Schema to Ready Dataset

Imagine an e-commerce database with 20 related tables: manually creating 10,000 records takes a week, and after integration, tests fail due to an edge case. You need realistic data — not just user_1 and [email protected]. To cover all scenarios, data must have correct distributions, boundary values, different characters, and relationships between tables. Manual creation is poor: either too little data or unrealistic.

Manual test data generation eats up hours or even days. You have to invent thousands of records, align foreign keys, remember nulls and long strings. AI-generated test data solves this in minutes — scaling from hundreds to millions of records without quality loss. We automate the process using LLMs and custom prompts: we create structured datasets from your DB schema, semantic content for your domain, and anomalous cases for negative tests. End-to-end — from schema to ready JSON/CSV. Our AI data generation delivers synthetic data with built-in data anonymization, ensuring PII protection. This solution accelerates test data automation and provides a complete dataset for tests.

Our approach delivers 10-15% edge case coverage. You get a dataset ready for production testing. All data is deterministic and validated on the fly. Contact us for a consultation — we'll prepare a demo dataset for your schema. We have 5+ years of AI/ML experience and have completed over 100 successful projects, ensuring enterprise-grade quality.

Problems Solved by AI Generation

Manual datasets contain 10-100 records, but you need 100,000+. AI scales without quality loss. Edge cases (empty strings, SQL injections, strings >1000 characters) are forgotten in 80% of manual projects. With several related tables, manual creation breaks referential integrity — the AI generator builds a topological order and pulls existing IDs. Production data contains personal information — DataAnonymizer replaces email, phone, passport with synthetic analogs, preserving structure.

How AI Ensures Data Realism

We use a combination: Faker AI integration for basic providers (names, addresses); LLM (GPT-4o) for semantic context — diagnoses via ICD-10, trade names of drugs, amounts with lognormal distribution; custom providers for domains: medical codes, IBAN. Model temperature of 0.8 gives diversity while maintaining plausibility. The generated semantic data accurately reflects real-world scenarios.

Why You Need Edge Cases

Edge data reveals bugs that don't reproduce on 'clean' data. Our generator includes 10-15% anomalous records in each batch. These are null, max int, special characters, SQL injections, long strings. In one e-commerce project, this helped find an integer overflow when processing high amounts — a bug that lived in production for 2 years. Testing edge cases is crucial for robust software.

How Generation of Related Data Works

The generator builds a topological order of dependencies. First, parent tables are created, then child tables — with substitution of existing FKs. This guarantees integrity of relationships. Request a consultation, and we'll configure the generator for your schema.

Process

1. Analysis — study the DB schema, business rules, test data requirements.
2. Design — configure Faker providers, write prompts for LLM, define distributions.
3. Implementation & Testing — generator code, integration with CI/CD (GitHub Actions, GitLab CI), dataset validation: schema, duplicates, FKs, edge cases.
4. Deployment — deliver script or API, train the team, provide documentation.

Comparison: Manual vs AI Generation

AI generator creates datasets 100x faster than manual effort. This streamlines database testing significantly.

What's Included in the Service

• Architecture of the AI generator for your DB schema.
• Custom Faker providers for the domain.
• PII anonymization module.
• CI/CD integration (containerization, Makefile).
• Documentation on use and configuration.
• Team training (1-2 hours online).
• 2 weeks of support after delivery.

Practical Case

Our client — an e-commerce platform with a test environment of 50,000 rows of production data (anonymized). After deploying the AI generator, the test dataset expanded to 500,000 records with realistic distributions. We found 3 performance bugs (slow queries) that did not reproduce on the small dataset. Edge data revealed an error in processing high amounts (integer overflow in old PHP code). Saved 4 hours of testing time per week — equivalent to $500 monthly savings. Our service typically costs $500 per month, so the savings directly offset the investment.

Generation Code

from faker import Faker
from langchain_openai import ChatOpenAI
from pydantic import BaseModel
import json
import random

fake = Faker("ru_RU")

class TestDataGenerator:
    SEMANTIC_PROMPT = """Generate {count} realistic records for the table.

Table schema:
{schema}

Domain context: {domain}

Requirements:
- Data should look realistic (not "test_value_1")
- Numeric values in reasonable ranges for the domain
- Dates distributed across different periods
- Statuses/categories unevenly distributed (realistic)
- Related fields consistent (e.g. city and region)
- Include 10-15% edge cases: min/max values, empty optional fields

Return a JSON array of {count} objects."""

    def __init__(self):
        self.llm = ChatOpenAI(model="gpt-4o", temperature=0.8)

    def generate_for_table(
        self,
        schema: dict,
        domain: str,
        count: int = 100
    ) -> list[dict]:
        result = self.llm.invoke(
            self.SEMANTIC_PROMPT.format(
                count=min(count, 50),
                schema=json.dumps(schema, ensure_ascii=False, indent=2),
                domain=domain
            )
        )
        batch = json.loads(result.content)

        if count > 50:
            all_data = batch
            while len(all_data) < count:
                more = self.generate_for_table(schema, domain, min(50, count - len(all_data)))
                all_data.extend(more)
            return all_data[:count]

        return batch

    def generate_boundary_cases(self, schema: dict) -> list[dict]:
        boundary_prompt = f"""Create edge and invalid test data for the schema.

Schema:
{json.dumps(schema, ensure_ascii=False, indent=2)}

Create 2-3 cases of each type:
1. Empty values (null, "", [])
2. Maximum values (max int, max string length)
3. Minimum values (min int, 0, negative)
4. Special characters: <, >, &, ', ", \n, \t, emoji 🎉
5. SQL-injection strings (for sanitization testing)
6. Very long strings (>1000 characters)
7. Wrong types (string instead of number etc.)

Return JSON with expected_behavior tag for each case."""

        return json.loads(
            self.llm.invoke(boundary_prompt).content
        )

class RelationalDataGenerator:
    def generate_dataset(
        self,
        schema: dict,
        counts: dict
    ) -> dict[str, list[dict]]:
        result = {}
        order = self._topological_sort(schema["tables"])

        for table_name in order:
            table_schema = next(t for t in schema["tables"] if t["name"] == table_name)
            count = counts.get(table_name, 10)

            fk_pools = {}
            for fk in table_schema.get("fk_relations", []):
                ref_table = fk["references_table"]
                if ref_table in result:
                    fk_pools[fk["column"]] = [r[fk["references_column"]] for r in result[ref_table]]

            records = self._generate_with_fk(table_schema, count, fk_pools)
            result[table_name] = records

        return result

    def _generate_with_fk(
        self,
        table_schema: dict,
        count: int,
        fk_pools: dict
    ) -> list[dict]:
        records = []
        for _ in range(count):
            record = {}
            for col in table_schema["columns"]:
                if col["name"] in fk_pools:
                    record[col["name"]] = random.choice(fk_pools[col["name"]])
                else:
                    record[col["name"]] = self._generate_field_value(col)
            records.append(record)
        return records

from faker.providers import BaseProvider

class MedicalDataProvider(BaseProvider):
    DIAGNOSES = ["J00", "K21.0", "I10", "E11", "M79.3"]
    MEDICATIONS = ["Амоксициллин 500мг", "Метформин 850мг", "Лизиноприл 10мг"]

    def diagnosis_code(self) -> str:
        return self.random_element(self.DIAGNOSES)

    def medication(self) -> str:
        return self.random_element(self.MEDICATIONS)

class FinanceDataProvider(BaseProvider):
    def iban(self) -> str:
        return f"BY{fake.numerify('##')}ALFA{fake.numerify('################')}"

    def transaction_amount(self) -> float:
        weights = [0.5, 0.3, 0.15, 0.05]
        ranges = [(1, 100), (100, 1000), (1000, 10000), (10000, 100000)]
        chosen_range = random.choices(ranges, weights=weights)[0]
        return round(random.uniform(*chosen_range), 2)

class DataAnonymizer:
    PII_FIELDS = {
        "email": lambda v: fake.email(),
        "phone": lambda v: fake.phone_number(),
        "name": lambda v: fake.name(),
        "inn": lambda v: fake.numerify("############"),
        "passport": lambda v: f"{fake.numerify('####')} {fake.numerify('######')}",
        "ip_address": lambda v: fake.ipv4_private(),
        "address": lambda v: fake.address(),
    }

    def anonymize_dataset(self, data: list[dict], pii_field_names: list[str]) -> list[dict]:
        result = []
        for record in data:
            anonymized = dict(record)
            for field in pii_field_names:
                if field in anonymized:
                    field_type = self._detect_field_type(field)
                    if field_type in self.PII_FIELDS:
                        anonymized[field] = self.PII_FIELDS[field_type](anonymized[field])
            result.append(anonymized)
        return result

Estimated Timeline

• Structured data generator: 1–2 weeks.
• With anonymization and relational generation: 3–4 weeks.

The cost is calculated individually. Get a consultation — we'll prepare a demo dataset for your DB schema. Contact us to evaluate your project. Starting from $500 per month for standard plans.

LLM Development: Fine-Tuning, RAG, Agents, and Production Deployment

Using GPT‑4 or Claude 3.5 Sonnet through a public API is not a solution — it's just a tool. When the requirement is to "make it like ChatGPT, but on our data," there is a real engineering challenge behind it: from prompt engineering to training a 70B model on your own infrastructure. End-to-end LLM solution development is a complex stack, and we have been doing it for over 5 years. During this time, we have completed over 20 projects in generative AI: from RAG systems for legal departments to custom support agents. Where exactly your task falls depends on data, latency requirements, budget, and how critical confidentiality is.

A typical situation: the client has already tried ChatGPT, but results are unstable — sometimes accurate, sometimes hallucinating. Or they need integration into a corporate portal while complying with security policies. Let's break down each layer of the stack in detail — from RAG to production deployment.

Why Do RAG Systems Break and How to Fix It?

RAG (Retrieval-Augmented Generation) looks simple: find relevant documents, put them in context, get an answer. In practice, it fails in several places.

Chunking without overlap. Classic mistake: chunk_size=512, overlap=0. If the answer lies across two chunks, retrieval won't find either with sufficient confidence. Solution: overlap 15–25% of chunk_size, or better yet, sentence-aware splitting with spaCy or NLTK instead of naive character splitting.

Poor embedder. text-embedding-ada-002 is good for general use, but on legal or medical texts, specialized models like E5-large-v2, BGE-M3, or fine-tuned sentence-transformers on domain data outperform it. Recall@5 differences can be 15–25%.

No re-ranking. Vector search optimizes for speed, not relevance. A cross-encoder re-ranker (ms-marco-MiniLM-L-6-v2, bge-reranker-large) after initial retrieval improves top-3 accuracy with acceptable latency (+50–150ms). This is often more impactful than improving the embedding model.

Hybrid search. Dense vectors alone work poorly on exact queries: names, SKUs, codes. BM25 (sparse) finds exact matches but misses semantics. Hybrid via RRF (Reciprocal Rank Fusion) is the optimal compromise. Qdrant, Weaviate, and pgvector 0.7+ support hybrid search natively.

When to Fine-Tune Instead of Prompt Engineering?

Prompt engineering solves ~70% of LLM adaptation tasks for a domain. The remaining 30% require fine-tuning. Three indicators: the model ignores a specific output format even with detailed prompting; the task requires deep knowledge of specialized vocabulary (medicine, law); you need to significantly reduce token costs by replacing a large model with a smaller specialized one.

LoRA and QLoRA are the standard for SFT. LoRA adds trainable low-rank matrices to attention layers. A typical configuration for Llama-3 8B: r=64, lora_alpha=128, target_modules=["q_proj","v_proj","k_proj","o_proj"] yields ~0.8% trainable parameters, training on one A100 40GB. QLoRA adds 4-bit quantization (NF4) and allows fine-tuning 70B models on two A100 40GB, though speed drops by half compared to bf16.

DPO instead of RLHF. Direct Preference Optimization requires only (chosen, rejected) pairs, not scalar reward signals. DPOTrainer from the trl library (Hugging Face) implements it in a few dozen lines.

Common mistake. A dataset of 500 examples, 5 epochs, validation loss 0.8 — seems fine. But on test, the model degrades on general instructions. Cause: catastrophic forgetting. Solution: add 10–20% general instruction-following examples (Alpaca, FLAN) to the training set to preserve original capabilities.

How to Choose a Base Model: 8B or 70B?

For most tasks, fine-tuning an 8B model is sufficient. 70B is needed when deep reasoning is required or the 8B baseline does not reach the required quality even after fine-tuning. Inference cost for Llama-3 8B via vLLM on A100 is efficient; the exact cost depends on volume.

What Does PagedAttention Bring to Production?

vLLM is the first choice for serving open-source models. PagedAttention is the key technical innovation: KV-cache is managed like virtual memory in an OS, without fragmentation. This yields 2–4x higher throughput compared to naive HuggingFace Transformers inference. The vLLM documentation confirms that continuous batching and PagedAttention are the standard for high-load LLM services.

Typical numbers on A100 80GB for Llama-3 8B (bf16): 400–600 req/s, P50 latency 200–400ms, P99 latency 600–900ms at concurrency 64. For 70B on two A100 with tensor parallelism: 80–120 req/s, P99 latency 1.5–2.5s. AWQ or GPTQ quantization reduces memory consumption by 2x with quality loss within 1–3%.

Multi-Agent Systems

Agents are LLMs with access to tools: search, code execution, API calls, database interaction. Common patterns:

• ReAct (Reason + Act): the model reasons → chooses a tool → observes the result → reasons again. LangChain and LlamaIndex implement it out of the box.
• Multi-agent orchestration: multiple specialized agents with a coordinator on top. Example: coordinator → researcher (search + summarization) → coder (code generation and execution) → critic (verification). Tools: AutoGen (Microsoft), CrewAI, custom implementation on LangGraph.

In production, agent systems are non-deterministic. Essential: guardrails, step limits, logging of each step, human-in-the-loop for critical actions.

How We Work: Stages, Timeline, Deliverables

What Is Included

We deliver:

• Technical documentation (model card, configs, deployment instructions)
• Access to infrastructure (code repository, trained weights)
• 1 month of post-deployment support (consultations, bug fixes)
• Customer team training (2–3 sessions on system operation)

Timeline: basic RAG prototype — 1–2 weeks. Fine-tuning with customer data — 3–6 weeks (including data preparation). Production system with monitoring and retraining — 2–4 months. Cost is calculated individually based on data volume, model complexity, and infrastructure requirements.

We guarantee the quality of the final model with performance benchmarks and ongoing monitoring. Our engineers have hands‑on experience with dozens of production LLM systems.

Want to evaluate your project? Leave a request — we will prepare a preliminary summary within 1–2 business days. Or get a consultation on choosing the approach: RAG, fine-tuning, or hybrid — we will tell you what works best for you. Contact us to discuss your LLM development needs. Schedule a free consultation today.