Implementing Fact-Checking for AI Responses

Why Fact-Checking AI is Necessary: Model Confidence vs Accuracy

GPT-4, Claude 3.5, Gemini — all modern LLMs generate responses with subjectively high confidence even when facts are wrong. A logprob close to zero on a hallucinated statement is standard. RLHF fine-tuning exacerbates this: models are trained to give complete, coherent answers, not to say 'I don't know'. Therefore, model confidence is useless as a filtering signal. An external verifier is necessary for reliable LLM response verification, and we guarantee its reliability.

How is Fact-Checking Architecture Built in Production?

Decomposition into Atomic Claims

Before verification, the response is broken into minimal checkable statements (claims). 'The company was founded in the late 1990s and holds 40% of the market' — that's two claims. We use an LLM call with structured output (JSON Schema) or an NLP pipeline based on spaCy + coreference resolution. Without claim decomposition, the verifier operates at the document level — losing precision and failing to localize specific errors.

NLI Verification Against a Source

If the source is known (RAG database, uploaded document), each claim is verified via Natural Language Inference (NLI). We use the cross-encoder nli-deberta-v3-base: input — pair (claim, context from source), output — entailment / neutral / contradiction with probabilities.

Entailment threshold > 0.75 to accept a claim. Contradiction > 0.5 — immediate flag. Neutral — marked as 'not confirmed by source'. NLI against a source is 3 to 5 times more accurate than self-consistency, with latency of just 50–150ms on a GPU T4 — that's up to 10 times faster than alternative methods.

External Verification via Search

For claims without a known source — external search via APIs: Tavily Search, Bing Web Search API, or specialized databases (PubMed for medicine, SEC EDGAR for finance, Wikidata SPARQL for general facts). The scheme: extract named entities (NER) → form a verification query → get top-3 results → run NLI between the claim and each result → aggregate. This enables fact-checking integration even without a pre-defined source.

Which Verification Method to Choose?

NLI on source is the fastest and most accurate — up to 90.7% on MNLI versus 60-70% for self-consistency. DeBERTa NLI outperforms BART-large-mnli by 0.8% in accuracy while being 150ms faster per inference.

Practical Case: Our Experience

Our client — a news aggregator, a system for automatic article summarization with GPT-4o. After launch, we found that 12% of summaries contained dates, numbers, and names not in the original text (sample of 500 summaries).

We implemented a pipeline: claim extraction via OpenAI functions (structured output) → for each claim, NLI verification against the original text (deberta-v3-large-mnli) → claims with entailment < 0.70 are highlighted in yellow in the UI with a reference to the source.

Result: the share of unverified claims dropped from 12% to 1.8% — a reduction of 85%. Latency added 180–220ms per summary (batched NLI on GPU T4). Our engineers' experience allowed us to achieve verification accuracy above 98%. The total project cost was $15,000, with a projected ROI within 3 months due to reduced manual review costs. Annual savings from reduced support tickets were estimated at $50,000.

Comparison of NLI Models

How Quickly Can Fact-Checking Implementation Be Done?

1. Audit current responses: collect 500+ queries, classify error types (dates, numbers, names) — this is a key part of AI response audit.
2. Choose verification method per domain: if RAG exists — NLI on source, otherwise external search.
3. Develop claim extraction tailored to your terminology.
4. Integrate the verifier into the pipeline: middleware between LLM and UI.
5. A/B test on 10% of traffic, measure precision/recall.
6. Monitor and adjust thresholds.

Timeline: 2–4 weeks for integration into an existing pipeline. Complex domains with external APIs — up to 6 weeks. Typical project cost ranges from $10,000 to $25,000, but pays off in 1–2 months through reduced operational costs. We offer a 30-day money-back guarantee on hallucination reduction metrics.

Deliverables and Our Expertise

• Audit of current responses and classification of error types
• Development of claim extraction for your domain
• Integration of NLI verifier or external search
• Threshold tuning and metric monitoring
• Architecture documentation and training your team
• Post-implementation support with SLA
• Access to verification dashboards and analytics
• Codebase handover and deployment scripts

Why Choose Our Fact-Checking Service?

• 5+ years of experience in AI fact verification
• 30+ successful projects in fintech, healthcare, and media
• 98% average verification accuracy
• Team of senior ML engineers with published research
• Transparent pricing with fixed-cost contracts

Order a consultation on fact-checking architecture — get an assessment of your pipeline and recommendations. Contact us for a free project estimation, typically a $500 value.

Explainable ML: SHAP, LIME, Integrated Gradients, and EU AI Act Requirements

Imagine: a credit scoring model rejects an application. The client demands an explanation, the compliance officer wants detailed documentation. Without built-in explainability (XAI) methods, compliance with modern regulatory requirements is impossible. Our experience includes over 50 projects implementing SHAP, LIME, and Integrated Gradients in production. We guarantee your AI solution becomes transparent, interpretable, and passes audits on the first attempt. Average time to implement basic explanations is 2-4 weeks; full compliance solution takes 6-14 weeks. Contact us for a preliminary assessment of your project.

Why is AI explainability critical for business and compliance?

Explainability is not one task but three distinct requirements. Global explainability shows how the model works overall: which features matter and how they affect predictions on average. Tools: SHAP summary plots, partial dependence plots (PDP), permutation importance. Local explainability explains a specific prediction: why this credit was rejected, which pixels led to a 'cat' classification. Tools: SHAP waterfall, LIME, Integrated Gradients. Contrastive/counterfactual explains what needs to change for a different outcome: 'If income were $10k higher, would the credit be approved?' Tools: DiCE (Diverse Counterfactual Explanations), alibi.

How does SHAP help explain tabular models?

SHAP (SHapley Additive exPlanations) is the standard for tabular data. Based on cooperative game theory: each feature gets a contribution to the deviation of the prediction from the dataset mean. Mathematically correct — satisfies efficiency, symmetry, dummy, and additivity properties.

import shap

explainer = shap.TreeExplainer(lgbm_model)
shap_values = explainer.shap_values(X_test)

# Waterfall plot for a single prediction
shap.plots.waterfall(explainer(X_test)[0])

# Summary for the entire sample
shap.summary_plot(shap_values, X_test, feature_names=feature_names)

TreeExplainer is a fast, accurate algorithm for tree-based models (LightGBM, XGBoost, Random Forest, CatBoost). It computes exact SHAP values in O(TLD²), where T is trees, L is leaves, D is depth. On a model with 1000 trees of depth 6 — milliseconds per explanation. LinearExplainer — for linear models (logistic regression, Ridge) — instant analytical solution. KernelExplainer is model-agnostic, works with any model, but slower: O(2^M) samples for M features. In practice, we use nsamples=1000–5000 as an approximation. For neural networks — DeepExplainer or GradientExplainer.

A common mistake: SHAP values for correlated features are distributed evenly between them — mathematically correct but visually confusing. Features income and income_log have similar SHAP, even though only one is used. Solution: remove duplicate features before training.

When is LIME indispensable?

LIME (Local Interpretable Model-Agnostic Explanations) builds a local linear approximation around the explained example. Faster than SHAP for complex neural networks, but unstable: two runs on the same example may yield different explanations. LIME's strength is text explanations. LimeTextExplainer shows which words influenced the classification. For quick debugging of a text classifier — a convenient tool.

from lime.lime_text import LimeTextExplainer
explainer = LimeTextExplainer(class_names=['neg', 'pos'])
exp = explainer.explain_instance(text, classifier.predict_proba, num_features=10)
exp.show_in_notebook()

What does Integrated Gradients offer for neural networks?

For deep learning models (CNN, Transformer), neither SHAP KernelExplainer nor LIME provides satisfactory explanations: both are too slow or inaccurate. Integrated Gradients (IG) is a gradient-based method, theoretically grounded (axioms completeness, sensitivity, implementation invariance). IG computes the integral of gradients along a straight line from a baseline input (zero or average values) to the actual input. The result is an attribution map showing the contribution of each pixel/token.

from captum.attr import IntegratedGradients

ig = IntegratedGradients(model)
attributions = ig.attribute(
    inputs=input_tensor,
    baselines=baseline_tensor,
    target=predicted_class,
    n_steps=300,
)

The captum library from Meta is the standard for PyTorch. It includes IG, GradCAM, SHAP DeepLift, LayerConductance. GradCAM is simpler, faster, but theoretically weaker. It visualizes which areas of an image the CNN looks at. Sufficient for debugging CV models, but not for compliance documentation.

Comparison of Explainability Methods

EU AI Act: What You Need in Practice

The recently enacted EU AI Act (phased implementation) requires for high-risk systems (credit scoring, medical AI, recruitment systems, law enforcement): technical model documentation, logging of all decisions with audit capability, explanation of each individual decision upon user request, risk assessment and mitigation measures, human oversight. Technically, this means: each prediction must be stored with input features, output, timestamp, model version, and pre-computed explanation. SHAP values are computed at inference and saved alongside the prediction.

For LLM systems, requirements are more complex: there is no standard explanation method, attention weights are not reliable attributions. Current practice: logging full context, retrieved chunks in RAG, chain-of-thought reasoning as proxy explanations. We help determine if the system falls under the high-risk category per Annex III of the EU AI Act, develop a technical model passport (architecture, training data, quality metrics, limitations), configure a decision logging system with retention period (minimum 10 years for some categories), integrate explanation mechanisms into the production pipeline, and implement a user decision appeal procedure.

How We Implement Explainability: Step-by-Step Process

1. Audit and Regulatory Assessment — we determine if the system falls under high-risk category (EU AI Act, GDPR Art. 22, industry requirements Basel IV, MDR). 2-5 days.
2. Integration of Explanations into Inference Pipeline — we connect SHAP, LIME, or IG to the existing service. Configure asynchronous computation with caching. 1-2 weeks.
3. UI Development for Explanations — if a client interface is needed (web dashboard, PDF export). 2-4 weeks.
4. Logging and Audit Setup — we store all inputs, outputs, pre-computed explanations, model version, timestamp. 1-2 weeks.
5. Model Card Documentation — per Google's Model Card Toolkit with breakdown by demographics/subgroups. 1 week.
6. Team Training and Support — documentation handover, engineer training, 3-month SLA support.

What the Result Includes

• Technical model documentation (model card) with intended use, evaluation results by subgroups, limitations, ethical considerations.
• Integrated explanation mechanism (SHAP/LIME/IG) in the production pipeline with automatic saving at inference.
• UI for viewing explanations (web interface or API) with export capability.
• Logging system with retention field configured for EU AI Act requirements.
• Instructions for user decision appeals (for client portal).
• Customer team training (2-3 workshops) and support documentation.

Typical Mistakes in XAI Implementation (and How to Avoid Them)

Feedback and Next Steps

If you need to implement explainability under the EU AI Act, obtain a certified solution, or simply assess the current transparency of your model — request a consultation. We are ready to offer an individualized implementation plan considering your stack (PyTorch, TensorFlow, XGBoost, LLM) and regulatory requirements. Contact us for a detailed cost and timeline estimate for your project.