Implementing SHAP/LIME for Model Explainability

Model XGBoost gives AUC 0.91 on validation. In production, unexpected predictions appear — high scores for obviously irrelevant objects. Feature importance from boosting itself shows top-10 features but does not explain a specific prediction. That particular object may get score 0.87 for non-obvious reasons — and we as engineers are obliged to answer. We implement SHAP and LIME for model explainability in production, and this is not just an audit — it is part of an end-to-end ML pipeline.

SHAP and LIME answer different versions of the question "why?". It is important to understand when to apply each method and where they break.

How do SHAP and LIME work?

SHAP (SHapley Additive exPlanations, Lundberg & Lee, 2017) is based on Shapley values from cooperative game theory. The idea: the contribution of each feature to the prediction is the average of its marginal influence across all possible feature coalitions. Key property: additivity. The sum of SHAP values of all features plus the base value (average model prediction) equals the specific prediction. This is a mathematically exact decomposition, not an approximation.

LIME (Locally Interpretable Model-agnostic Explanations, Ribeiro et al., 2016) works differently: a random cloud of perturbations is generated around the object, the black-box model predicts each perturbed instance, and then a simple interpretable model (linear regression or decision tree) is trained on that cloud. LIME is stochastic, so in production we fix the seed and use num_samples=5000+.

What problems do SHAP and LIME solve?

1. Failures of feature importance. Built-in feature importance in XGBoost shows a global picture but does not explain a single case. SHAP solves this with deterministic decomposition.
2. Black box for business. Regulators require explanations for each decision. TreeSHAP provides transparency in acceptable time.
3. Model drift without signal. SHAP values logged to ClickHouse allow tracking changes in feature influence earlier than metric drops.

TreeSHAP — why architectural specialization matters

For tree-based models (XGBoost, LightGBM, CatBoost, sklearn RandomForest) there is TreeSHAP — an algorithm with polynomial complexity O(TLD²). This is orders of magnitude faster than naive KernelSHAP.

import shap
import xgboost as xgb

model = xgb.XGBClassifier()
model.fit(X_train, y_train)

explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)

# Waterfall plot for a specific prediction
shap.plots.waterfall(explainer(X_test)[0])

# Summary plot — global importance
shap.summary_plot(shap_values, X_test)

In practice, TreeSHAP on LightGBM with 500 trees processes 10,000 examples in 2–3 seconds on CPU. Quite acceptable for batch inference.

Why LIME is sometimes better than SHAP?

• The model is not supported by TreeSHAP and KernelSHAP is too slow.
• You need an explanation in terms of "superpixels" for images or word highlighting for texts.
• A quick prototype without deep mathematics is required.

But remember: LIME is not deterministic. With different random_state, explanations for the same object can differ. In production we use a fixed seed and num_samples=5000+.

Method comparison

Typical problems and solutions

Integration into production ML pipeline

Explanations are needed not only for audit — they are part of the operational pipeline.

Case study: client — an insurance company, calculating insurance premiums (LightGBM, 120 features). Requirement: an agent must explain over the phone why a premium is high. Solution: TreeSHAP in the inference API. For each prediction, return the top-3 features with the highest SHAP values + an automatic text template: "Your premium is above average due to: vehicle age (+12%), registration region (+8%), claims history (+6%)". Latency overhead: 35ms for TreeSHAP with average inference 18ms — acceptable.

Monitoring: SHAP values are logged to ClickHouse. Once a week we aggregate — drift in SHAP value distribution signals feature drift earlier than AUC drop.

Limitations to be aware of

SHAP ≠ causality. A high SHAP value for a feature means correlation with the prediction, not causation. "Feature X influences the prediction" ≠ "changing X will change the outcome in reality".

Multicollinearity breaks interpretation. If two features are correlated (r > 0.8), SHAP splits their influence arbitrarily. Correlation analysis is needed when interpreting.

For LLMs — both methods give rough estimates. Attention weights are often more informative for generation tasks, but are also not a strict proxy for importance.

What is included in our work

• Model and data analysis: select the appropriate method — TreeSHAP, KernelSHAP, LIME — considering architecture and latency requirements.
• Explainer module development: integration into the existing inference API.
• Report generation: waterfall plots, summary plots, automatic text templates.
• Monitoring: logging SHAP values to ClickHouse, building drift dashboards.
• Team training: documentation, workshop for engineers and business users.

Timelines and cost

Timelines: from 1 week for basic integration of one method to 3–4 weeks for a full pipeline with monitoring and dashboards. Cost is calculated individually for each project. Contact us for a preliminary assessment — we will tell you what results you will get.

Our experience: over 50 projects in explainable AI, 5 years in the market, certified ML engineers. We guarantee transparency and post-implementation support.

Request a consultation — we'll help make your model explainable and compliant with regulatory requirements.

Explainable ML: SHAP, LIME, Integrated Gradients, and EU AI Act Requirements

Imagine: a credit scoring model rejects an application. The client demands an explanation, the compliance officer wants detailed documentation. Without built-in explainability (XAI) methods, compliance with modern regulatory requirements is impossible. Our experience includes over 50 projects implementing SHAP, LIME, and Integrated Gradients in production. We guarantee your AI solution becomes transparent, interpretable, and passes audits on the first attempt. Average time to implement basic explanations is 2-4 weeks; full compliance solution takes 6-14 weeks. Contact us for a preliminary assessment of your project.

Why is AI explainability critical for business and compliance?

Explainability is not one task but three distinct requirements. Global explainability shows how the model works overall: which features matter and how they affect predictions on average. Tools: SHAP summary plots, partial dependence plots (PDP), permutation importance. Local explainability explains a specific prediction: why this credit was rejected, which pixels led to a 'cat' classification. Tools: SHAP waterfall, LIME, Integrated Gradients. Contrastive/counterfactual explains what needs to change for a different outcome: 'If income were $10k higher, would the credit be approved?' Tools: DiCE (Diverse Counterfactual Explanations), alibi.

How does SHAP help explain tabular models?

SHAP (SHapley Additive exPlanations) is the standard for tabular data. Based on cooperative game theory: each feature gets a contribution to the deviation of the prediction from the dataset mean. Mathematically correct — satisfies efficiency, symmetry, dummy, and additivity properties.

import shap

explainer = shap.TreeExplainer(lgbm_model)
shap_values = explainer.shap_values(X_test)

# Waterfall plot for a single prediction
shap.plots.waterfall(explainer(X_test)[0])

# Summary for the entire sample
shap.summary_plot(shap_values, X_test, feature_names=feature_names)

TreeExplainer is a fast, accurate algorithm for tree-based models (LightGBM, XGBoost, Random Forest, CatBoost). It computes exact SHAP values in O(TLD²), where T is trees, L is leaves, D is depth. On a model with 1000 trees of depth 6 — milliseconds per explanation. LinearExplainer — for linear models (logistic regression, Ridge) — instant analytical solution. KernelExplainer is model-agnostic, works with any model, but slower: O(2^M) samples for M features. In practice, we use nsamples=1000–5000 as an approximation. For neural networks — DeepExplainer or GradientExplainer.

A common mistake: SHAP values for correlated features are distributed evenly between them — mathematically correct but visually confusing. Features income and income_log have similar SHAP, even though only one is used. Solution: remove duplicate features before training.

When is LIME indispensable?

LIME (Local Interpretable Model-Agnostic Explanations) builds a local linear approximation around the explained example. Faster than SHAP for complex neural networks, but unstable: two runs on the same example may yield different explanations. LIME's strength is text explanations. LimeTextExplainer shows which words influenced the classification. For quick debugging of a text classifier — a convenient tool.

from lime.lime_text import LimeTextExplainer
explainer = LimeTextExplainer(class_names=['neg', 'pos'])
exp = explainer.explain_instance(text, classifier.predict_proba, num_features=10)
exp.show_in_notebook()

What does Integrated Gradients offer for neural networks?

For deep learning models (CNN, Transformer), neither SHAP KernelExplainer nor LIME provides satisfactory explanations: both are too slow or inaccurate. Integrated Gradients (IG) is a gradient-based method, theoretically grounded (axioms completeness, sensitivity, implementation invariance). IG computes the integral of gradients along a straight line from a baseline input (zero or average values) to the actual input. The result is an attribution map showing the contribution of each pixel/token.

from captum.attr import IntegratedGradients

ig = IntegratedGradients(model)
attributions = ig.attribute(
    inputs=input_tensor,
    baselines=baseline_tensor,
    target=predicted_class,
    n_steps=300,
)

The captum library from Meta is the standard for PyTorch. It includes IG, GradCAM, SHAP DeepLift, LayerConductance. GradCAM is simpler, faster, but theoretically weaker. It visualizes which areas of an image the CNN looks at. Sufficient for debugging CV models, but not for compliance documentation.

Comparison of Explainability Methods

EU AI Act: What You Need in Practice

The recently enacted EU AI Act (phased implementation) requires for high-risk systems (credit scoring, medical AI, recruitment systems, law enforcement): technical model documentation, logging of all decisions with audit capability, explanation of each individual decision upon user request, risk assessment and mitigation measures, human oversight. Technically, this means: each prediction must be stored with input features, output, timestamp, model version, and pre-computed explanation. SHAP values are computed at inference and saved alongside the prediction.

For LLM systems, requirements are more complex: there is no standard explanation method, attention weights are not reliable attributions. Current practice: logging full context, retrieved chunks in RAG, chain-of-thought reasoning as proxy explanations. We help determine if the system falls under the high-risk category per Annex III of the EU AI Act, develop a technical model passport (architecture, training data, quality metrics, limitations), configure a decision logging system with retention period (minimum 10 years for some categories), integrate explanation mechanisms into the production pipeline, and implement a user decision appeal procedure.

How We Implement Explainability: Step-by-Step Process

1. Audit and Regulatory Assessment — we determine if the system falls under high-risk category (EU AI Act, GDPR Art. 22, industry requirements Basel IV, MDR). 2-5 days.
2. Integration of Explanations into Inference Pipeline — we connect SHAP, LIME, or IG to the existing service. Configure asynchronous computation with caching. 1-2 weeks.
3. UI Development for Explanations — if a client interface is needed (web dashboard, PDF export). 2-4 weeks.
4. Logging and Audit Setup — we store all inputs, outputs, pre-computed explanations, model version, timestamp. 1-2 weeks.
5. Model Card Documentation — per Google's Model Card Toolkit with breakdown by demographics/subgroups. 1 week.
6. Team Training and Support — documentation handover, engineer training, 3-month SLA support.

What the Result Includes

• Technical model documentation (model card) with intended use, evaluation results by subgroups, limitations, ethical considerations.
• Integrated explanation mechanism (SHAP/LIME/IG) in the production pipeline with automatic saving at inference.
• UI for viewing explanations (web interface or API) with export capability.
• Logging system with retention field configured for EU AI Act requirements.
• Instructions for user decision appeals (for client portal).
• Customer team training (2-3 workshops) and support documentation.

Typical Mistakes in XAI Implementation (and How to Avoid Them)

Feedback and Next Steps

If you need to implement explainability under the EU AI Act, obtain a certified solution, or simply assess the current transparency of your model — request a consultation. We are ready to offer an individualized implementation plan considering your stack (PyTorch, TensorFlow, XGBoost, LLM) and regulatory requirements. Contact us for a detailed cost and timeline estimate for your project.