AI Auto-Generation of Integration Tests

AI Auto-Generation of Integration Tests

Integration tests often become a bottleneck in the CI pipeline: they are either missing or failing due to fragile fixtures. We solve this with AI-generation that analyzes service architecture, database schema, and OpenAPI specifications. The result — ready-to-use pytest tests that find real bugs without producing false positives. Regression time drops to 4 hours instead of three days, and testing costs are reduced by up to 40%.

What Problems Does AI Test Generation Solve?

Manual writing of integration tests takes 2–3 days of a QA engineer's time per release. Meanwhile, 80% of test code is boilerplate: repetitive CRUD checks, API calls with different parameters, and test data setup. An AI generator takes over the routine, freeing the team to focus on complex cases.

Incomplete coverage is a common pain in microservice architectures. Manually, it's easy to miss a chain like "order → payment → notification → warehouse." AI enumerates all contract combinations and generates tests for every integration path, including edge cases. Coverage increases 4–6 times compared to the manual approach.

Fragile tests — fixtures tied to production data — break with every schema change. We generate isolated fixtures using factory_boy with transactions and rollback, ensuring stability and repeatability.

How Does AI Generate Integration Tests?

We use a combination of LangChain and ChatOpenAI (GPT-4o with temperature 0.1 for determinism). Generation proceeds in multiple passes: first schema and contract analysis, then code creation, then syntax validation and execution on a mock environment. The final test code conforms to pytest standards and is ready for repository inclusion.

Case from our practice: an e-commerce platform with 15 microservices. Problem: integration testing was done manually before each release (2 QA × 3 days). We generated 180 integration tests for critical paths: order → payment → notification → inventory update. Of these 180 tests, 23 failed on the first run — they found real bugs: errors in currency conversion handling, duplicate events in the queue, incorrect status for partial payments. After fixes, integration coverage rose from 15% to 85%, and regression time dropped from 3 days to 4 hours. The client reduced testing costs by 40%.

from langchain_openai import ChatOpenAI
import json
from pathlib import Path

class IntegrationTestGenerator:
    INTEGRATION_PROMPT = """Create an integration test for component interaction.

Components and their contracts:
{components}

Database schema:
{db_schema}

Integration scenario:
{scenario}

Requirements:
- pytest + SQLAlchemy for database work (use transactions with rollback)
- httpx.AsyncClient for HTTP calls
- pytest-asyncio for async tests
- Test database via pytest fixture (not production!)
- Check not only HTTP status but also DB state after the operation
- Use factory_boy or pytest-factoryboy for test data

Return the complete test code with fixtures."""

    def __init__(self):
        self.llm = ChatOpenAI(model="gpt-4o", temperature=0.1)

    def generate_db_integration_tests(
        self,
        model_code: str,
        repository_code: str,
        db_schema: str
    ) -> str:
        prompt = f"""Create pytest integration tests for Repository and Model.

SQLAlchemy Model:
```python
{model_code}

Repository:

{repository_code}

DDL schema:

{db_schema}

Create tests:

1. CRUD operations (create, read, update, delete)
2. Filtering and sorting
3. Transactions (successful commit, rollback on error)
4. Unique constraints (attempt to insert duplicate)
5. Foreign key constraints
6. Pagination (if present in Repository)

Fixtures:

• db_session: SQLAlchemy session with rollback after each test
• test_data factories via factory_boy

Return the test code.""" return self.llm.invoke(prompt).content

def generate_service_integration_tests(
    self,
    service_a_spec: dict,
    service_b_spec: dict,
    interaction_patterns: list[str]
) -> str:
    """Generates tests for service-to-service interaction"""
    prompt = f"""Create pytest integration tests for service interaction.

Service A (client):

• Base URL: {service_a_spec['base_url']}
• Calls: {json.dumps(service_a_spec['calls'], ensure_ascii=False)}

Service B (server):

• Endpoints: {json.dumps(service_b_spec.get('endpoints', []), ensure_ascii=False)}

Interaction patterns: {chr(10).join(f"- {p}" for p in interaction_patterns)}

Use:

• pytest + respx for mocking HTTP responses of service B
• Tests for retry logic (what happens on timeout of service B)
• Tests for circuit breaker (if present)
• Tests for correct handling of 4xx/5xx from service B

Return test code with fixtures.""" return self.llm.invoke(prompt).content

### Generating Fixtures and Test Data

```python
class TestDataGenerator:
    FACTORY_PROMPT = """Create factory_boy factories for models.

SQLAlchemy models:
{models_code}

Create:
1. Factory for each model
2. SubFactory for related objects
3. Trait for specific states (e.g., expired_user, admin_user)
4. Batch creation via factory.build_batch

Return the factories code."""

    async def generate_factories(self, models_code: str) -> str:
        result = await self.llm.ainvoke(
            self.FACTORY_PROMPT.format(models_code=models_code)
        )
        return result.content

    async def generate_fixtures_from_schema(self, schema: dict) -> str:
        """Creates pytest fixtures from DB schema"""
        prompt = f"""Create pytest fixtures for test database.

Schema: {json.dumps(schema, ensure_ascii=False, indent=2)}

Fixtures needed:
- engine: SQLAlchemy engine to test DB (PostgreSQL via pytest-postgresql)
- db_session: session with rollback after each test
- Fixtures for each table: minimal valid object
- seeded_db: database with initial data for e2e tests

Use scope='function' for db_session, scope='session' for engine.
Return Python code."""
        return (await self.llm.ainvoke(prompt)).content

For message queues (RabbitMQ/Kafka), we use testcontainers-python, generating tests for send, process, and dead letter queue. Detailed examples are provided during individual consultation.

Why AI Generation Is More Reliable Than Manual Writing?

AI tests are created 5x faster and deliver coverage of 80–90% versus 10–20% with manual effort. They verify all integration paths, including edge cases and error handling that QA often miss. Moreover, generated tests are independent of production data and stable under schema changes.

What's Included in the Work?

Comparison of Manual vs AI Approach

Quality Guarantee for Generated Tests

Each generated test undergoes a two-stage verification: syntax analysis with pytest --co and execution in an isolated staging environment with real dependencies. We measure module-level coverage and fix all false positives. Experience shows that 90% of tests pass on the first run. The remaining 10% are edge cases that we manually refine. As a result, clients receive a stable test suite ready for integration into a pytest pipeline.

Estimated Timelines

• Basic package (DB + services): from 3 to 5 weeks.
• Full package (DB + services + queues + testcontainers): from 5 to 7 weeks.

Pricing is individual after a project audit. Contact us to assess your project. Fill out the form on our website, and we'll estimate the workload within 1 business day.

AI generation never misses an endpoint, never forgets the dead letter queue, and never makes mistakes in typed assertions. Integration test coverage increases from 10–20% to 80–90% in a single generation cycle. Our engineers guarantee that the generated tests pass code review and are ready for CI execution.

Request AI test generation implementation now and get an engineer consultation. Contact us — we'll help accelerate regression testing and reduce costs.

LLM Development: Fine-Tuning, RAG, Agents, and Production Deployment

Using GPT‑4 or Claude 3.5 Sonnet through a public API is not a solution — it's just a tool. When the requirement is to "make it like ChatGPT, but on our data," there is a real engineering challenge behind it: from prompt engineering to training a 70B model on your own infrastructure. End-to-end LLM solution development is a complex stack, and we have been doing it for over 5 years. During this time, we have completed over 20 projects in generative AI: from RAG systems for legal departments to custom support agents. Where exactly your task falls depends on data, latency requirements, budget, and how critical confidentiality is.

A typical situation: the client has already tried ChatGPT, but results are unstable — sometimes accurate, sometimes hallucinating. Or they need integration into a corporate portal while complying with security policies. Let's break down each layer of the stack in detail — from RAG to production deployment.

Why Do RAG Systems Break and How to Fix It?

RAG (Retrieval-Augmented Generation) looks simple: find relevant documents, put them in context, get an answer. In practice, it fails in several places.

Chunking without overlap. Classic mistake: chunk_size=512, overlap=0. If the answer lies across two chunks, retrieval won't find either with sufficient confidence. Solution: overlap 15–25% of chunk_size, or better yet, sentence-aware splitting with spaCy or NLTK instead of naive character splitting.

Poor embedder. text-embedding-ada-002 is good for general use, but on legal or medical texts, specialized models like E5-large-v2, BGE-M3, or fine-tuned sentence-transformers on domain data outperform it. Recall@5 differences can be 15–25%.

No re-ranking. Vector search optimizes for speed, not relevance. A cross-encoder re-ranker (ms-marco-MiniLM-L-6-v2, bge-reranker-large) after initial retrieval improves top-3 accuracy with acceptable latency (+50–150ms). This is often more impactful than improving the embedding model.

Hybrid search. Dense vectors alone work poorly on exact queries: names, SKUs, codes. BM25 (sparse) finds exact matches but misses semantics. Hybrid via RRF (Reciprocal Rank Fusion) is the optimal compromise. Qdrant, Weaviate, and pgvector 0.7+ support hybrid search natively.

When to Fine-Tune Instead of Prompt Engineering?

Prompt engineering solves ~70% of LLM adaptation tasks for a domain. The remaining 30% require fine-tuning. Three indicators: the model ignores a specific output format even with detailed prompting; the task requires deep knowledge of specialized vocabulary (medicine, law); you need to significantly reduce token costs by replacing a large model with a smaller specialized one.

LoRA and QLoRA are the standard for SFT. LoRA adds trainable low-rank matrices to attention layers. A typical configuration for Llama-3 8B: r=64, lora_alpha=128, target_modules=["q_proj","v_proj","k_proj","o_proj"] yields ~0.8% trainable parameters, training on one A100 40GB. QLoRA adds 4-bit quantization (NF4) and allows fine-tuning 70B models on two A100 40GB, though speed drops by half compared to bf16.

DPO instead of RLHF. Direct Preference Optimization requires only (chosen, rejected) pairs, not scalar reward signals. DPOTrainer from the trl library (Hugging Face) implements it in a few dozen lines.

Common mistake. A dataset of 500 examples, 5 epochs, validation loss 0.8 — seems fine. But on test, the model degrades on general instructions. Cause: catastrophic forgetting. Solution: add 10–20% general instruction-following examples (Alpaca, FLAN) to the training set to preserve original capabilities.

How to Choose a Base Model: 8B or 70B?

For most tasks, fine-tuning an 8B model is sufficient. 70B is needed when deep reasoning is required or the 8B baseline does not reach the required quality even after fine-tuning. Inference cost for Llama-3 8B via vLLM on A100 is efficient; the exact cost depends on volume.

What Does PagedAttention Bring to Production?

vLLM is the first choice for serving open-source models. PagedAttention is the key technical innovation: KV-cache is managed like virtual memory in an OS, without fragmentation. This yields 2–4x higher throughput compared to naive HuggingFace Transformers inference. The vLLM documentation confirms that continuous batching and PagedAttention are the standard for high-load LLM services.

Typical numbers on A100 80GB for Llama-3 8B (bf16): 400–600 req/s, P50 latency 200–400ms, P99 latency 600–900ms at concurrency 64. For 70B on two A100 with tensor parallelism: 80–120 req/s, P99 latency 1.5–2.5s. AWQ or GPTQ quantization reduces memory consumption by 2x with quality loss within 1–3%.

Multi-Agent Systems

Agents are LLMs with access to tools: search, code execution, API calls, database interaction. Common patterns:

• ReAct (Reason + Act): the model reasons → chooses a tool → observes the result → reasons again. LangChain and LlamaIndex implement it out of the box.
• Multi-agent orchestration: multiple specialized agents with a coordinator on top. Example: coordinator → researcher (search + summarization) → coder (code generation and execution) → critic (verification). Tools: AutoGen (Microsoft), CrewAI, custom implementation on LangGraph.

In production, agent systems are non-deterministic. Essential: guardrails, step limits, logging of each step, human-in-the-loop for critical actions.

How We Work: Stages, Timeline, Deliverables

What Is Included

We deliver:

• Technical documentation (model card, configs, deployment instructions)
• Access to infrastructure (code repository, trained weights)
• 1 month of post-deployment support (consultations, bug fixes)
• Customer team training (2–3 sessions on system operation)

Timeline: basic RAG prototype — 1–2 weeks. Fine-tuning with customer data — 3–6 weeks (including data preparation). Production system with monitoring and retraining — 2–4 months. Cost is calculated individually based on data volume, model complexity, and infrastructure requirements.

We guarantee the quality of the final model with performance benchmarks and ongoing monitoring. Our engineers have hands‑on experience with dozens of production LLM systems.

Want to evaluate your project? Leave a request — we will prepare a preliminary summary within 1–2 business days. Or get a consultation on choosing the approach: RAG, fine-tuning, or hybrid — we will tell you what works best for you. Contact us to discuss your LLM development needs. Schedule a free consultation today.