AI-Powered Federated Search Across Multiple Sources

Picture this: a call center operator handling a customer request has to switch between Confluence, Jira, SharePoint, and Outlook to gather information. In an enterprise context, data is scattered across dozens of systems—Confluence for documentation, Jira for tasks, SharePoint for files, corporate email, CRM, external databases—each with its own search interface. Employees spend up to 40% of their time searching and switching between systems. Federated search with a single search bar and unified ranking cuts that time by 2-3x. We design and implement such systems turnkey: from requirements analysis to deployment on client infrastructure. Our experience: 5+ years in AI/ML and Enterprise AI, 15+ federated search projects delivered. Typical pilot cost ranges from $15,000 to $25,000, fully recouped within 6 months through productivity gains. We guarantee SLA on latency (p99 < 1 sec) and result quality (NDCG@10 > 0.85).

Why Federated Search Is Harder Than Regular Search

The reason is source heterogeneity. BM25 scores from Elasticsearch are incomparable with vector proximity from Qdrant. Response times vary: Confluence may respond in 100 ms, while an external API might take 2 seconds. An asynchronous architecture with timeouts, score normalization, deduplication, and ranking is required. These are non-trivial MLOps challenges. Our multi-source search solution integrates seamlessly with your existing systems, and our result ranking model ensures the most relevant documents appear first.

How Hybrid Architecture Works

In most enterprise projects, we use a hybrid approach: Confluence, SharePoint, Jira—through a centralized index; external APIs and real-time data—through query-time federation.

Hybrid architecture achieves latency of 300-800 ms, which is 2-5x faster than pure query-time federation.

import asyncio
from typing import Protocol, runtime_checkable
from dataclasses import dataclass

@dataclass
class SearchResult:
    source: str
    doc_id: str
    title: str
    snippet: str
    url: str
    score: float
    metadata: dict

@runtime_checkable
class SearchConnector(Protocol):
    async def search(self, query: str, filters: dict, limit: int) -> list[SearchResult]:
        ...

class FederatedSearchOrchestrator:
    def __init__(self, connectors: dict[str, SearchConnector]):
        self.connectors = connectors
        self.merger = ResultMerger()

    async def search(
        self,
        query: str,
        sources: list[str] = None,
        filters: dict = None,
        limit: int = 10
    ) -> list[SearchResult]:
        active_connectors = {
            name: conn for name, conn in self.connectors.items()
            if sources is None or name in sources
        }

        # Parallel query to all sources with timeout
        tasks = {
            name: asyncio.create_task(
                asyncio.wait_for(
                    conn.search(query, filters or {}, limit * 2),
                    timeout=3.0  # don't wait for slow sources longer than 3 sec
                )
            )
            for name, conn in active_connectors.items()
        }

        results_by_source = {}
        for name, task in tasks.items():
            try:
                results_by_source[name] = await task
            except asyncio.TimeoutError:
                results_by_source[name] = []  # source did not respond
            except Exception as e:
                results_by_source[name] = []

        return self.merger.merge_and_rank(results_by_source, limit)

How Deduplication and Re-ranking Are Done

The main technical challenge in federated search is normalizing scores from different sources. BM25 scores from Elasticsearch are incomparable with cosine similarity from Qdrant. We use a CrossEncoder for final ranking and cosine similarity of embeddings for deduplication. More about CrossEncoder can be found in the documentation.

from sentence_transformers import CrossEncoder
import numpy as np

class ResultMerger:
    def __init__(self):
        self.reranker = CrossEncoder(
            "cross-encoder/ms-marco-MiniLM-L-6-v2",
            max_length=512
        )

    def merge_and_rank(
        self,
        results_by_source: dict[str, list[SearchResult]],
        limit: int
    ) -> list[SearchResult]:
        all_results = []
        for source, results in results_by_source.items():
            all_results.extend(results)

        if not all_results:
            return []

        # Deduplication by content (cosine similarity of embeddings)
        all_results = self._deduplicate(all_results, threshold=0.92)

        # Normalize scores within each source (min-max)
        for source in results_by_source:
            source_results = [r for r in all_results if r.source == source]
            if len(source_results) > 1:
                scores = [r.score for r in source_results]
                min_s, max_s = min(scores), max(scores)
                for r in source_results:
                    r.score = (r.score - min_s) / (max_s - min_s + 1e-9)

        return sorted(all_results, key=lambda x: x.score, reverse=True)[:limit]

Comparison of deduplication methods for document deduplication:

Connectors for Specific Sources

class ConfluenceConnector:
    async def search(self, query: str, filters: dict, limit: int) -> list[SearchResult]:
        # Hybrid search: vector store (Qdrant) for semantics
        # + Confluence REST API for freshness
        vector_results = await self.vector_store.asimilarity_search(query, k=limit)
        return [self._to_result(r) for r in vector_results]

class JiraConnector:
    async def search(self, query: str, filters: dict, limit: int) -> list[SearchResult]:
        # JQL with text search
        jql = f'text ~ "{query}" ORDER BY updated DESC'
        if filters.get("project"):
            jql = f'project = {filters["project"]} AND ' + jql
        issues = self.jira.search_issues(jql, maxResults=limit)
        return [self._issue_to_result(i) for i in issues]

class EmailConnector:
    async def search(self, query: str, filters: dict, limit: int) -> list[SearchResult]:
        # Search via Microsoft Graph API or IMAP
        results = await self.graph_client.search_messages(
            query=query,
            top=limit,
            select=["subject", "bodyPreview", "from", "receivedDateTime"]
        )
        return [self._email_to_result(r) for r in results]

Case study: An insurance company with 600 employees. 6 sources: SharePoint (180K documents), Jira, Outlook, internal document management system, precedent database, regulatory database (external API). Before implementation, a call center operator switched between 4-5 tabs when handling a client request. After: one search interface with results from all sources. Average query handling time: 4.2 min → 1.8 min. The regulatory database source via query-time federation—latency 800 ms, acceptable for this scenario. Time savings: over 100 hours per month for the department, allowing a reduction of 2 FTEs. Annual savings from reduced search time typically range from $50,000 to $100,000 per department.

How It Works: Step by Step

1. Connect sources: We develop adapters for your enterprise systems (Confluence, Jira, SharePoint, databases, etc.) using their APIs.
2. Configure score normalization: We set up MinMax scaling and embedding-based deduplication to unify results.
3. Deploy reranker model: A CrossEncoder model re-ranks the top candidates for final relevance.
4. Set up monitoring: MLOps dashboards track latency, result quality (NDCG), and embedding drift.

Personalization of Sources by Role

Different roles see relevant sources by default. For query routing, we use LangChain Expression Language (LCEL), simplifying rule configuration.

ROLE_SOURCE_CONFIG = {
    "developer": ["jira", "confluence", "gitlab", "stackoverflow-internal"],
    "hr": ["confluence", "email", "hr-system", "orgchart"],
    "lawyer": ["contracts-db", "sharepoint", "email", "regulations"],
    "support": ["confluence", "jira", "email", "crm", "knowledge-base"],
}

Deliverables: What Is Included in the Implementation

• Prototype: Working federated search for 2-3 sources within 4-6 weeks.
• Documentation: Architectural documentation and API specification.
• Connectors: Custom adapters for each source.
• Ranking module: Deduplication and ranking module (CrossEncoder).
• Monitoring: Quality and latency dashboard (MLOps).
• Training: Team training and 1 month of post-launch support.

Timelines: 2-3 sources, pilot: 4-6 weeks; full federation of 6-8 sources: 3-4 months. Cost is calculated individually and is recouped through reduced employee labor costs.

How We Guarantee Quality

With 5+ years in enterprise AI and 15+ successful deployments, our team ensures robust E-A-T. We use certified models (CrossEncoder on RoBERTa), conduct A/B testing of ranking before deployment. We implement monitoring for embedding quality drift and automatic recalculation of normalization. We guarantee NDCG@10 not lower than 0.85 after tuning.

Assess your project—contact us for a consultation. Order a pilot in 4-6 weeks, and we will show how federated search accelerates your employees' work. Get demo access to a working prototype.

LLM Development: Fine-Tuning, RAG, Agents, and Production Deployment

Using GPT‑4 or Claude 3.5 Sonnet through a public API is not a solution — it's just a tool. When the requirement is to "make it like ChatGPT, but on our data," there is a real engineering challenge behind it: from prompt engineering to training a 70B model on your own infrastructure. End-to-end LLM solution development is a complex stack, and we have been doing it for over 5 years. During this time, we have completed over 20 projects in generative AI: from RAG systems for legal departments to custom support agents. Where exactly your task falls depends on data, latency requirements, budget, and how critical confidentiality is.

A typical situation: the client has already tried ChatGPT, but results are unstable — sometimes accurate, sometimes hallucinating. Or they need integration into a corporate portal while complying with security policies. Let's break down each layer of the stack in detail — from RAG to production deployment.

Why Do RAG Systems Break and How to Fix It?

RAG (Retrieval-Augmented Generation) looks simple: find relevant documents, put them in context, get an answer. In practice, it fails in several places.

Chunking without overlap. Classic mistake: chunk_size=512, overlap=0. If the answer lies across two chunks, retrieval won't find either with sufficient confidence. Solution: overlap 15–25% of chunk_size, or better yet, sentence-aware splitting with spaCy or NLTK instead of naive character splitting.

Poor embedder. text-embedding-ada-002 is good for general use, but on legal or medical texts, specialized models like E5-large-v2, BGE-M3, or fine-tuned sentence-transformers on domain data outperform it. Recall@5 differences can be 15–25%.

No re-ranking. Vector search optimizes for speed, not relevance. A cross-encoder re-ranker (ms-marco-MiniLM-L-6-v2, bge-reranker-large) after initial retrieval improves top-3 accuracy with acceptable latency (+50–150ms). This is often more impactful than improving the embedding model.

Hybrid search. Dense vectors alone work poorly on exact queries: names, SKUs, codes. BM25 (sparse) finds exact matches but misses semantics. Hybrid via RRF (Reciprocal Rank Fusion) is the optimal compromise. Qdrant, Weaviate, and pgvector 0.7+ support hybrid search natively.

When to Fine-Tune Instead of Prompt Engineering?

Prompt engineering solves ~70% of LLM adaptation tasks for a domain. The remaining 30% require fine-tuning. Three indicators: the model ignores a specific output format even with detailed prompting; the task requires deep knowledge of specialized vocabulary (medicine, law); you need to significantly reduce token costs by replacing a large model with a smaller specialized one.

LoRA and QLoRA are the standard for SFT. LoRA adds trainable low-rank matrices to attention layers. A typical configuration for Llama-3 8B: r=64, lora_alpha=128, target_modules=["q_proj","v_proj","k_proj","o_proj"] yields ~0.8% trainable parameters, training on one A100 40GB. QLoRA adds 4-bit quantization (NF4) and allows fine-tuning 70B models on two A100 40GB, though speed drops by half compared to bf16.

DPO instead of RLHF. Direct Preference Optimization requires only (chosen, rejected) pairs, not scalar reward signals. DPOTrainer from the trl library (Hugging Face) implements it in a few dozen lines.

Common mistake. A dataset of 500 examples, 5 epochs, validation loss 0.8 — seems fine. But on test, the model degrades on general instructions. Cause: catastrophic forgetting. Solution: add 10–20% general instruction-following examples (Alpaca, FLAN) to the training set to preserve original capabilities.

How to Choose a Base Model: 8B or 70B?

For most tasks, fine-tuning an 8B model is sufficient. 70B is needed when deep reasoning is required or the 8B baseline does not reach the required quality even after fine-tuning. Inference cost for Llama-3 8B via vLLM on A100 is efficient; the exact cost depends on volume.

What Does PagedAttention Bring to Production?

vLLM is the first choice for serving open-source models. PagedAttention is the key technical innovation: KV-cache is managed like virtual memory in an OS, without fragmentation. This yields 2–4x higher throughput compared to naive HuggingFace Transformers inference. The vLLM documentation confirms that continuous batching and PagedAttention are the standard for high-load LLM services.

Typical numbers on A100 80GB for Llama-3 8B (bf16): 400–600 req/s, P50 latency 200–400ms, P99 latency 600–900ms at concurrency 64. For 70B on two A100 with tensor parallelism: 80–120 req/s, P99 latency 1.5–2.5s. AWQ or GPTQ quantization reduces memory consumption by 2x with quality loss within 1–3%.

Multi-Agent Systems

Agents are LLMs with access to tools: search, code execution, API calls, database interaction. Common patterns:

• ReAct (Reason + Act): the model reasons → chooses a tool → observes the result → reasons again. LangChain and LlamaIndex implement it out of the box.
• Multi-agent orchestration: multiple specialized agents with a coordinator on top. Example: coordinator → researcher (search + summarization) → coder (code generation and execution) → critic (verification). Tools: AutoGen (Microsoft), CrewAI, custom implementation on LangGraph.

In production, agent systems are non-deterministic. Essential: guardrails, step limits, logging of each step, human-in-the-loop for critical actions.

How We Work: Stages, Timeline, Deliverables

What Is Included

We deliver:

• Technical documentation (model card, configs, deployment instructions)
• Access to infrastructure (code repository, trained weights)
• 1 month of post-deployment support (consultations, bug fixes)
• Customer team training (2–3 sessions on system operation)

Timeline: basic RAG prototype — 1–2 weeks. Fine-tuning with customer data — 3–6 weeks (including data preparation). Production system with monitoring and retraining — 2–4 months. Cost is calculated individually based on data volume, model complexity, and infrastructure requirements.

We guarantee the quality of the final model with performance benchmarks and ongoing monitoring. Our engineers have hands‑on experience with dozens of production LLM systems.

Want to evaluate your project? Leave a request — we will prepare a preliminary summary within 1–2 business days. Or get a consultation on choosing the approach: RAG, fine-tuning, or hybrid — we will tell you what works best for you. Contact us to discuss your LLM development needs. Schedule a free consultation today.