AI System for Identifying Knowledge Gaps in Organizations

Consider this: when an employee spends 30 minutes searching for an answer, and the knowledge base stays silent — it's not laziness. It's a hole in the corporate knowledge corpus. A typical company with 500 employees generates over 10,000 search queries per day, of which 15% yield no results. Each such query means lost time and risk of errors. New employee onboarding drags on for weeks, support is flooded with repetitive questions, and the corporate knowledge base turns into a graveyard of outdated instructions. We developed an AI system that automatically detects these gaps and assesses their priority so you can close them before they become critical.

Knowledge Gap Analysis (KGA) is a method that identifies the mismatch between what employees know and what they need for their work. Our AI for corporate training automatically finds gaps. This missing expertise detection is a core feature. The system analyzes the knowledge corpus, support tickets, and search behavior. It pinpoints areas where information is insufficient. This is not just an audit — it's a tool that automatically prioritizes and prepares an action plan. This knowledge audit automation replaces manual work, saving up to 80% of analyst time.

Gap analysis is the foundation for this approach.

Sources for Gap Analysis

The system collects data from five channels. Each provides a different slice of the problem.

Search queries in the internal knowledge base — a query with zero results clearly indicates a gap. If an answer exists but nobody clicks it, the content is bad. Support tickets — repeated questions signal missing documentation. Questions in corporate chats — a valuable unstructured source. Onboarding completion rates show where new employees get stuck. The system uses RAG for knowledge management and ML models for topic and sentiment classification. Using LLM for training gap analysis ensures accuracy in coverage assessment.

How AI Determines a Critical Gap?

We use a scoring model that considers query frequency, number of unique users, trends, and business impact. Our system is 10x faster than manual analysis, completing in hours what takes weeks. High-priority gaps land first in the content plan. Here's an implementation example:

class KnowledgeGapDetector:
    def analyze_search_logs(self, search_logs: list[SearchLog]) -> list[KnowledgeGap]:
        gaps = []

        # Группируем zero-result запросы по семантической близости
        zero_results = [log for log in search_logs if log.result_count == 0]
        clusters = cluster_queries(zero_results)

        for cluster in clusters:
            gaps.append(KnowledgeGap(
                topic=cluster.representative_query,
                evidence_queries=cluster.queries[:10],
                frequency=len(cluster.queries),
                unique_users=len({q.user_id for q in cluster.queries}),
                gap_type="missing_content",
                priority=self.calculate_priority(cluster)
            ))

        # Запросы с результатом, но низким engagement
        low_engagement = [
            log for log in search_logs
            if log.result_count > 0 and log.clicked_result is None
        ]
        clusters_low = cluster_queries(low_engagement)
        for cluster in clusters_low:
            gaps.append(KnowledgeGap(
                topic=cluster.representative_query,
                frequency=len(cluster.queries),
                gap_type="poor_quality_content",
                existing_articles=find_related_articles(cluster.representative_query),
                priority=self.calculate_priority(cluster)
            ))

        return sorted(gaps, key=lambda g: g.priority, reverse=True)

    def calculate_priority(self, cluster) -> float:
        # Приоритет = частота × количество уникальных пользователей × срочность
        urgency_bonus = 2.0 if cluster.has_recent_spike() else 1.0
        return (cluster.frequency * len(cluster.unique_users) * urgency_bonus) ** 0.5

This algorithm reduced onboarding time by 40% in one project within three months. In another case, the system helped a 500-employee company save up to 2,000,000 rubles per year on onboarding and cut support load by 25%, with a typical ROI of 5x within the first year.

Step-by-Step Analysis Process

1. Data collection — from search logs, ticketing system, chats (Slack, Teams), LMS.
2. Query clustering — group queries by semantic similarity (using 1536-dim embeddings).
3. Gap scoring — compute priority using the formula above.
4. Coverage assessment — via LLM (GPT-4) we check how thoroughly each topic is covered in the knowledge base.
5. Report generation — a detailed plan with deadlines.

Knowledge Base Coverage Analysis

For each business topic, the system evaluates coverage via an LLM. The code iterates over topics and parses the model's response.

class CoverageAnalyzer:
    def assess_coverage(
        self,
        required_topics: list[str],
        knowledge_base: KnowledgeBase
    ) -> CoverageReport:

        coverage = {}
        for topic in required_topics:
            articles = knowledge_base.search(topic, top_k=5)
            if not articles:
                coverage[topic] = CoverageStatus(level=0.0, status="missing")
                continue

            coverage_score = llm.parse(f"""Оцени покрытие темы '{topic}' по найденным статьям.

Статьи:
{format_articles(articles)}

Оцени:
- Полнота (0-1): насколько полно тема раскрыта
- Актуальность (0-1): насколько информация свежая
- Практичность (0-1): есть ли примеры, инструкции
- Что не хватает: конкретные аспекты темы без покрытия""",
                response_format=CoverageScore
            )
            coverage[topic] = CoverageStatus(
                level=coverage_score.overall,
                status="adequate" if coverage_score.overall > 0.7 else "insufficient",
                gaps=coverage_score.missing_aspects
            )

        return CoverageReport(
            total_topics=len(required_topics),
            well_covered=[t for t, c in coverage.items() if c.level > 0.7],
            gaps=[t for t, c in coverage.items() if c.level <= 0.7],
            coverage_map=coverage
        )

The result is a report detailing each topic: coverage percentage, status, and list of missing aspects.

How the Content Plan Is Generated?

Based on the analysis, the system automatically generates a content plan. For each gap, it proposes a structure, title, and author (via Expertise Locator — searching for specialists by tag in the corporate network). Tasks are created in Jira/Confluence with deadlines that depend on query frequency: the more frequent the query, the sooner the gap must be closed. We ensure the plan is realistic and balanced in effort.

What's Included in the Work

We deliver a turnkey result: from audit to ready plan.

Total timeline: from 2 weeks to 2 months. We'll give an accurate estimate after a brief.

Inefficiency of Manual Knowledge Audits

Manual analysis requires weeks of an analyst's time. It's subjective and quickly outdated. With 5+ years on the market and 30+ successful implementations, our AI system processes thousands of queries in minutes, objectively ranks gaps, and updates automatically. It closes gaps 3x faster than traditional methods. Our track record includes 50+ satisfied clients. We guarantee quality and confidentiality.

Request a demo to see how AI gap analysis reduces onboarding time by 40% and saves up to 2,000,000 rubles per year. Get a consultation right now.

NLP Development: Text Classification, NER, Embeddings, and Information Extraction

We often receive a task: process 50,000 support tickets — currently all manual. Dataset — 3,000 labeled examples, 12 categories, imbalance: one category occupies 40% of the sample, three at 1-2% each. Baseline accuracy — 78%. Sounds decent until you look at recall for rare classes: 0.31, 0.44, 0.28. These classes — complaints and churn threats — are most important to the business.

This is a typical NLP development project. The problem is not the algorithm but that accuracy is the wrong metric. Our experience across 30+ projects shows: we start by analyzing business metrics and only then choose the model.

Why accuracy is not the right metric for rare classes?

Accuracy ignores imbalance. If the "churn" class appears in 2% of cases, the model can predict "all good" and get 98% accuracy — but the business loses clients. Solution: F1 macro (averaged over all classes) or weighted F1. For NER — strict entity F1 (exact matches only). We guarantee: after choosing the correct metric, model quality becomes measurable and predictable.

Text Classification: From BERT to Distillation

BERT-like models are the standard for classification. ruBERT-base or ruBERT-large from DeepPavlov for Russian. multilingual-e5-large — for multiple languages in one pipeline. XLM-RoBERTa-large — a strong multilingual backbone.

Fine-tuning for classification: add a classification head on top of the [CLS] token, train for 3-5 epochs with lr=2e-5, weight decay=0.01. For imbalance — weighted CrossEntropyLoss or focal loss with gamma=2.0. Contact us — we will show a code snippet.

Imbalance case study. Dataset — 3,000 examples, imbalance 1:20. Solution: class_weight via sklearn + CrossEntropyLoss. Additionally — augmentation of rare classes via backtranslation (ru→en→ru through MarianMT). Recall for rare classes rose from 0.31 to 0.67 with a slight drop in accuracy (76%→74%). Full NLP development end-to-end took 3 weeks.

Distillation for production. BERT-large gives F1 0.89, but inference on CPU — 180ms. Distillation into DistilBERT or ruBERT-tiny2 reduces latency to 25ms with F1 0.84. Export to ONNX Runtime provides an additional 1.5-2x speedup. DistilBERT achieves 7x lower latency than BERT-large with only a 5% drop in macro F1 – a typical production trade-off.

How to choose between BERT and LLM for your task?

For most classification and extraction tasks, BERT-sized models offer the best trade-off between cost and performance. Shift to LLMs only when the task demands generation, complex reasoning, or zero-shot generalization.

NER: Named Entity Recognition

NER — extracting persons, organizations, locations, dates, amounts, document numbers. For general categories (PER, ORG, LOC), pre-trained models work well. For specialized ones (medical terms, legal concepts) — fine-tuning is needed.

Data annotation. The main cost of an NER project. For a quality model — 500-2,000 labeled sentences per entity type. Tools: Label Studio (open source) or Prodigy (by spaCy creators). IOB2 format — standard.

Architecture. Token classification on top of BERT: each token gets a label (B-PER, I-PER, O). spaCy 3.x with transformer pipeline — a convenient production choice.

Nested entities. Standard IOB models cannot handle nested entities (organization inside an address). For such tasks — span-based NER: SpanBERT or SpERT. More complex but correct.

Post-processing is mandatory. The model predicts tokens — normalized entities are needed. Date — dateparser. Amounts — regex + validation. Names — deduplication via rapidfuzz. Included in our standard delivery.

Sentiment Analysis and Opinion Mining

Binary classification positive/negative works out of the box with BERT. Complexity — aspect-based sentiment analysis (ABSA): "the restaurant has good food but terrible service." For ABSA: aspect extraction (NER) + sentiment per aspect. Joint models BERT-for-ABSA — quality on Russian data is lower due to dataset scarcity. RuSentiment, SentiRuEval — main resources.

For production with simple positive/negative/neutral: distil models are enough. Three classes, balanced dataset, 2,000+ examples — F1 macro 0.82-0.87 in 1-2 days.

Text Summarization

Extractive summarization (select sentences) — TextRank or BM25 without training. Fast, no hallucinations. Good for long documents.

Abstractive (generates new text) — seq2seq: mT5, mBART, FRED-T5, ruT5-large. For production via LLM API (GPT-4, Claude) — often the best cost/quality/speed trade-off.

Embeddings: Vector Representations of Text

Embeddings are the foundation of semantic search, deduplication, clustering, RAG. Quality critically affects downstream tasks.

Models. E5-large-v2, BGE-M3, multilingual-e5-large — strong multilingual embedders. sentence-transformers/paraphrase-multilingual-mpnet-base-v2 — fast option. For Russian: ru-en-RoSBERTa (Skoltech) performs well on semantic textual similarity.

Embedding quality evaluation uses the MTEB benchmark as standard. But top results on MTEB don't guarantee success on a domain dataset — we build domain-specific eval.

Fine-tuning embeddings. If standard models don't give the required Recall@k — contrastive learning on domain pairs with MultipleNegativesRankingLoss. How to perform this for domain data:

1. Collect 500–2,000 semantically similar pairs from your domain.
2. Apply MultipleNegativesRankingLoss with a batch size of 32–64.
3. Train for 1–3 epochs using AdamW (lr=2e-5).
4. Evaluate Recall@k on a held-out domain test set.

This approach yields a 5–15% improvement in Recall@k in practice.

Dimensionality and storage. E5-large: 1024 dim, float32 — 4KB per vector. For 10M documents — 40GB. Quantization int8 reduces to 10GB. FAISS IVF_PQ — more compact but with losses. Included in our deployment recommendations.

Information Extraction

Structured extraction is a frequent task. Examples: key contract terms, technical characteristics, dates and amounts from invoices.

1. Regex + rule-based. For INN, OGRN, amounts, dates — more reliable than neural networks. No data required.
2. NER + post-processing. For variable formats.
3. LLM with structured output. GPT‑4 / Claude with JSON schema — for complex documents. Cost: minimal per document. For 10k+ documents/day — we calculate the economics.

We guarantee a hybrid: regex/NER for typical fields + LLM for edge cases. Our guarantee is backed by years of production experience and more than 30 projects.

Work Stages

Prototype on existing data — 1-3 weeks. Production system with CI/CD — 1.5–2.5 months. Cost is calculated individually — get a consultation for a project estimate.

What's Included

• Model and pipeline architecture documentation
• Access to the model via REST API (FastAPI + ONNX)
• Client team training (2-hour webinar + Q&A)
• Accuracy guarantee on the agreed test set
• Months of post-delivery support (bug fixes, adaptation to new data)

Our Experience

Years of NLP projects from classification to RAG systems. The team includes ML engineers experienced with Hugging Face, spaCy, LangChain, MLOps. We use vLLM, Kubeflow, Weights & Biases — a production stack, not toys. Contact us to evaluate your NLP project within two days — request a free consultation on your text processing pipeline.