Part 4: Explainable AI (XAI): Using SHAP and LIME to Interpret Complex Models

Introduction

In the previous episodes of this series, we explored interactive dashboards, statistical rigor, and techniques for handling imbalanced data. Now, we arrive at a critical challenge in modern machine learning: explainability. As models grow more complex—ensemble methods, deep neural networks, gradient boosting machines—they often become “black boxes” that deliver impressive predictions but offer little insight into why they made those decisions.

Explainable AI (XAI) addresses this gap. Whether you’re satisfying regulatory requirements (like GDPR’s “right to explanation”), debugging model behavior, or building stakeholder trust, understanding how your model arrives at predictions is essential. In this final episode, we’ll dive deep into two of the most popular XAI frameworks: SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations). We’ll implement both on a Kaggle dataset, visualize their outputs interactively, and discuss when to use global vs. local interpretability.

Why Explainability Matters

Regulatory Compliance

The EU’s General Data Protection Regulation (GDPR) grants individuals the “right to explanation” for automated decisions that significantly affect them (Article 22). If your credit application is denied by an AI, you deserve to know why. In healthcare, finance, and hiring, model transparency isn’t optional—it’s legally mandated.

Debugging and Validation

Even if your model achieves 95% accuracy, it might be relying on spurious correlations (recall Part 2’s discussion on statistical rigor). SHAP and LIME can reveal if your house price predictor is using “number of photos in the listing” instead of actual square footage—a classic data leakage issue.

Building Trust

Stakeholders—executives, clinicians, end users—are more likely to adopt AI systems when they understand the reasoning. A doctor won’t trust a cancer diagnosis from a model that can’t explain which features drove the prediction.

SHAP: SHapley Additive exPlanations

Game Theory Foundation

SHAP values are rooted in cooperative game theory, specifically the Shapley value from 1953. Imagine a coalition game where players (features) collaborate to produce a payout (the model’s prediction). The Shapley value fairly distributes the total payout among players based on their marginal contributions across all possible coalitions.

For a feature $i$ and prediction $f(x)$, the SHAP value $\phi_i$ is:

$$\phi_i = \sum_{S \subseteq F \setminus \{i\}} \frac{|S|! (|F| – |S| – 1)!}{|F|!} \big[ f_{S \cup \{i\}}(x_{S \cup \{i\}}) – f_S(x_S) \big]$$

Where:
– $F$ is the set of all features
– $S$ is a subset of features not including $i$
– $f_S(x_S)$ is the model’s prediction using only features in $S$
– $|S|!$ and $(|F| – |S| – 1)!$ are factorials weighting each coalition
– $|F|!$ normalizes the sum

The bracketed term measures the marginal contribution of feature $i$ when added to coalition $S$. By averaging over all possible coalitions, SHAP ensures fairness: features that contribute more receive higher values.

TreeSHAP and KernelSHAP

Computing exact Shapley values requires $2^{|F|}$ evaluations—infeasible for high-dimensional data. The shap library offers optimized algorithms:

• TreeSHAP: Polynomial-time algorithm for tree-based models (XGBoost, LightGBM, Random Forest). Exploits tree structure to compute exact SHAP values efficiently.
• KernelSHAP: Model-agnostic approximation using weighted linear regression. Works for any model but slower than TreeSHAP.

Key Visualizations

1. Summary Plot: Global view showing feature importance and impact direction (positive/negative).
2. Dependence Plot: Scatter plot of SHAP values vs. feature values, revealing relationships.
3. Force Plot: Waterfall-style visualization showing how features push prediction above/below baseline.
4. Waterfall Plot: Similar to force plot but vertically stacked for easier reading.

LIME: Local Interpretable Model-agnostic Explanations

Local Surrogate Models

LIME takes a different approach: instead of global explanations, it focuses on local interpretability. For a single prediction, LIME perturbs the input, observes how predictions change, and fits a simple interpretable model (like linear regression) in that local neighborhood.

The algorithm:
1. Perturb: Generate synthetic samples near the instance of interest by randomly toggling features.
2. Predict: Run the black-box model on perturbed samples.
3. Weight: Assign higher weights to samples closer to the original instance (using exponential kernel).
4. Fit: Train a linear model on weighted samples to approximate local behavior.
5. Explain: Interpret the linear model’s coefficients as feature importances.

Tabular, Text, and Image LIME

• Tabular: Perturbs numerical features by sampling from training distribution; categorical features by toggling.
• Text: Perturbs by removing words; explanations highlight important words/phrases.
• Image: Perturbs by masking superpixels; explanations show which image regions matter.

LIME is model-agnostic—it works with any classifier or regressor, even proprietary APIs where you only have input-output access.

Practical Tutorial: House Prices Dataset

Let’s train an XGBoost model on Kaggle’s famous House Prices dataset and explain predictions with both SHAP and LIME.

Step 1: Data Preparation

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from xgboost import XGBRegressor
import shap
import lime
import lime.lime_tabular
import matplotlib.pyplot as plt
import plotly.graph_objects as go

# Load data (assuming train.csv is downloaded from Kaggle)
df = pd.read_csv('train.csv')

# Select features for simplicity (you can expand this)
features = ['OverallQual', 'GrLivArea', 'GarageCars', 'TotalBsmtSF', 
            'FullBath', 'YearBuilt', 'YearRemodAdd', 'Fireplaces',
            'BsmtQual', 'KitchenQual', 'Neighborhood']

df_subset = df[features + ['SalePrice']].copy()

# Handle missing values
df_subset['TotalBsmtSF'].fillna(0, inplace=True)
df_subset['BsmtQual'].fillna('NA', inplace=True)
df_subset['GarageCars'].fillna(0, inplace=True)

# Encode categorical variables
le_bsmt = LabelEncoder()
le_kitchen = LabelEncoder()
le_neighborhood = LabelEncoder()

df_subset['BsmtQual'] = le_bsmt.fit_transform(df_subset['BsmtQual'])
df_subset['KitchenQual'] = le_kitchen.fit_transform(df_subset['KitchenQual'])
df_subset['Neighborhood'] = le_neighborhood.fit_transform(df_subset['Neighborhood'])

X = df_subset.drop('SalePrice', axis=1)
y = df_subset['SalePrice']

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

Step 2: Train XGBoost Model

# Train model
model = XGBRegressor(
    n_estimators=100,
    max_depth=5,
    learning_rate=0.1,
    random_state=42
)

model.fit(X_train, y_train)

# Evaluate
train_score = model.score(X_train, y_train)
test_score = model.score(X_test, y_test)

print(f"Train R²: {train_score:.4f}")
print(f"Test R²: {test_score:.4f}")

Step 3: SHAP Analysis

# Create TreeExplainer for XGBoost
explainer_shap = shap.TreeExplainer(model)
shap_values = explainer_shap(X_test)

# Summary plot: global feature importance
shap.summary_plot(shap_values, X_test, feature_names=X.columns.tolist())

The summary plot shows each feature’s impact distribution. For example, GrLivArea (above-grade living area) likely shows high positive SHAP values for larger homes—intuitively, more square footage increases price.

Step 4: SHAP Dependence Plot

# Dependence plot for GrLivArea
shap.dependence_plot(
    'GrLivArea', 
    shap_values.values, 
    X_test, 
    interaction_index='OverallQual'
)

This scatter plot reveals how GrLivArea‘s SHAP value changes with its actual value, colored by OverallQual (interaction feature). You might see that larger homes have higher SHAP values, but this effect is amplified for high-quality homes.

Step 5: SHAP Waterfall Plot for Single Prediction

# Explain first test instance
instance_idx = 0
shap.waterfall_plot(shap_values[instance_idx])

The waterfall plot starts at the base value (mean prediction across training set) and shows how each feature pushes the prediction up or down, arriving at the final predicted price.

Step 6: Interactive SHAP Force Plot with Plotly

SHAP’s built-in force_plot uses JavaScript. Let’s create a custom Plotly version for embedding in dashboards:

def plotly_force_plot(base_value, shap_vals, feature_vals, feature_names):
    """
    Custom interactive force plot using Plotly.

    Args:
        base_value: Expected value (baseline)
        shap_vals: SHAP values for single instance
        feature_vals: Feature values for single instance
        feature_names: List of feature names
    """
    # Sort by absolute SHAP value
    indices = np.argsort(np.abs(shap_vals))[::-1]
    sorted_shap = shap_vals[indices]
    sorted_names = [feature_names[i] for i in indices]
    sorted_vals = feature_vals[indices]

    # Cumulative sum for positioning
    cumsum = np.insert(np.cumsum(sorted_shap), 0, 0) + base_value

    # Create bar chart
    colors = ['red' if s < 0 else 'blue' for s in sorted_shap]

    fig = go.Figure()

    for i, (name, shap_val, feat_val, cum_start, cum_end, color) in enumerate(
        zip(sorted_names, sorted_shap, sorted_vals, cumsum[:-1], cumsum[1:], colors)
    ):
        fig.add_trace(go.Bar(
            x=[shap_val],
            y=[i],
            orientation='h',
            name=f"{name}={feat_val:.2f}",
            marker_color=color,
            text=f"{shap_val:+.0f}",
            textposition='inside',
            hovertemplate=f"<b>{name}</b><br>Value: {feat_val:.2f}<br>SHAP: {shap_val:+.0f}<extra></extra>"
        ))

    # Add baseline and prediction markers
    fig.add_vline(x=base_value, line_dash="dash", line_color="gray", 
                  annotation_text=f"Base: ${base_value:,.0f}")
    fig.add_vline(x=cumsum[-1], line_dash="dash", line_color="green", 
                  annotation_text=f"Prediction: ${cumsum[-1]:,.0f}")

    fig.update_layout(
        title="Interactive SHAP Force Plot",
        xaxis_title="Price Impact ($)",
        yaxis_title="",
        yaxis=dict(showticklabels=False),
        showlegend=True,
        height=400,
        barmode='relative'
    )

    fig.show()

# Example usage
instance_idx = 0
plotly_force_plot(
    base_value=explainer_shap.expected_value,
    shap_vals=shap_values.values[instance_idx],
    feature_vals=X_test.iloc[instance_idx].values,
    feature_names=X.columns.tolist()
)

This Plotly version is fully interactive: hover to see exact SHAP values, zoom, and export as PNG. Perfect for embedding in Streamlit dashboards (recall Part 1!).

Step 7: LIME Explanation

# Create LIME explainer
explainer_lime = lime.lime_tabular.LimeTabularExplainer(
    training_data=X_train.values,
    feature_names=X.columns.tolist(),
    mode='regression',
    random_state=42
)

# Explain first test instance
instance_idx = 0
exp = explainer_lime.explain_instance(
    data_row=X_test.iloc[instance_idx].values,
    predict_fn=model.predict,
    num_features=10
)

# Visualize
exp.show_in_notebook(show_table=True)

LIME’s output shows the top features as a horizontal bar chart with their coefficients in the local linear model. For example:

GrLivArea > 2000: +15,000
OverallQual = 8: +12,000
Neighborhood = 15: -5,000

These are local approximations—they explain this specific house but may not generalize globally.

Step 8: SHAP vs. LIME Side-by-Side Comparison

import matplotlib.pyplot as plt

# Get LIME weights as dict
lime_weights = dict(exp.as_list())

# Get SHAP values for same instance
shap_instance = shap_values[instance_idx].values
shap_dict = dict(zip(X.columns, shap_instance))

# Plot side by side
fig, axes = plt.subplots(1, 2, figsize=(14, 6))

# LIME
features_lime = list(lime_weights.keys())
values_lime = list(lime_weights.values())
axes[0].barh(features_lime, values_lime, color=['red' if v < 0 else 'blue' for v in values_lime])
axes[0].set_title('LIME Explanation')
axes[0].set_xlabel('Coefficient')

# SHAP
features_shap = X.columns.tolist()
values_shap = [shap_dict[f] for f in features_shap]
axes[1].barh(features_shap, values_shap, color=['red' if v < 0 else 'blue' for v in values_shap])
axes[1].set_title('SHAP Explanation')
axes[1].set_xlabel('SHAP Value')

plt.tight_layout()
plt.show()

You’ll notice similarities—both methods likely rank GrLivArea and OverallQual highly—but LIME’s coefficients are local approximations while SHAP’s values are theoretically grounded in game theory.

Global vs. Local Interpretability

When to Use Global Explanations

Global interpretability reveals overall model behavior across the entire dataset. Use global methods when:

• Model debugging: Identifying which features the model relies on globally.
• Feature engineering: Deciding which features to keep, remove, or transform.
• Regulatory compliance: Demonstrating that the model doesn’t use protected attributes (e.g., race, gender) as primary drivers.

SHAP summary plots and permutation importance are excellent global tools.

When to Use Local Explanations

Local interpretability explains individual predictions. Use local methods when:

• Justifying decisions: Explaining to a loan applicant why they were denied.
• Error analysis: Investigating why the model failed on specific instances.
• Building trust: Showing stakeholders that the model’s reasoning aligns with domain knowledge for sample cases.

SHAP waterfall plots, LIME, and force plots excel here.

Hybrid Approach

In practice, combine both:
1. Start with global SHAP summary to understand overall feature importance.
2. Drill down with local LIME/SHAP on outliers, errors, or high-stakes predictions.
3. Validate that local explanations align with domain expertise.

Emerging XAI Methods

While SHAP and LIME dominate, newer techniques are gaining traction:

Integrated Gradients

Developed by Google for neural networks, Integrated Gradients computes feature attributions by integrating gradients along a straight path from a baseline input to the actual input. Unlike vanilla gradients (which can be noisy), integrated gradients satisfy axioms like sensitivity and implementation invariance.

Formula:

$$\text{IG}_i(x) = (x_i – x_i') \int_{\alpha=0}^{1} \frac{\partial f(x' + \alpha (x – x'))}{\partial x_i} d\alpha$$

Where:
– $x$ is the input instance
– $x'$ is the baseline (e.g., all zeros or mean feature values)
– $\frac{\partial f}{\partial x_i}$ is the gradient of the model output w.r.t. feature $i$
– The integral accumulates gradients along the interpolation path

Use case: Deep learning models for text and images.

Attention Visualization

For Transformer-based models (BERT, GPT, Vision Transformers), attention weights provide built-in interpretability. By visualizing which tokens the model attends to, you can see what information flows through the network.

Caveat: Attention is not explanation—high attention doesn’t always mean high importance (see “Attention is not Explanation” debates). Combine with gradient-based methods for robust analysis.

Concept-Based Explanations

TCAV (Testing with Concept Activation Vectors) explains models in terms of human-friendly concepts rather than raw features. For example, instead of “pixel 342 is important,” TCAV might say “the model detects ‘stripedness’ for zebra classification.”

How it works:
1. Train linear classifiers to separate concept examples (e.g., striped vs. non-striped images).
2. Compute directional derivatives to measure sensitivity to these concepts.
3. Report concept importance scores.

Use case: Image classifiers where pixel-level explanations are too granular.

Best Practices for XAI

1. Don’t Trust Explanations Blindly: Adversarial examples can manipulate explanations. Always validate with domain expertise.
2. Use Multiple Methods: SHAP and LIME can disagree. Triangulate with permutation importance, partial dependence plots, etc.
3. Explain What Matters: For high-stakes decisions (loan approvals, medical diagnoses), local explanations are critical. For model development, global insights suffice.
4. Interactive Visualizations: Static plots are useful, but interactive dashboards (Plotly, Streamlit) engage stakeholders better. Recall Part 1’s emphasis on interactivity!
5. Document Assumptions: LIME assumes linearity in local neighborhoods; SHAP assumes feature independence in some variants (KernelSHAP). Know the limitations.

Code Repository and Reproducibility

To reproduce this tutorial:

# Install dependencies
pip install pandas numpy scikit-learn xgboost shap lime plotly matplotlib

# Download Kaggle House Prices dataset
# (requires Kaggle API: kaggle competitions download -c house-prices-advanced-regression-techniques)

# Run the notebook
jupyter notebook explainable_ai_tutorial.ipynb

All code is self-contained and uses fixed random seeds for reproducibility.

Conclusion

Explainable AI transforms black-box models into transparent, trustworthy systems. SHAP provides theoretically rigorous, game-theory-based attributions that work globally and locally. LIME offers model-agnostic, easy-to-understand local explanations through surrogate linear models. By combining both—and visualizing results interactively with tools like Plotly—you can debug models, satisfy regulators, and build stakeholder confidence.

As we close this four-part series on The Art of Data Storytelling, remember that insights are only valuable if they’re understood. Whether you’re building interactive dashboards (Part 1), ensuring statistical rigor (Part 2), handling imbalanced data (Part 3), or explaining complex models (Part 4), the goal remains the same: make data speak clearly and compellingly.

Emerging methods like Integrated Gradients and concept-based explanations push XAI further, but SHAP and LIME remain the workhorses of production systems. Master these tools, validate their outputs, and always prioritize interpretability alongside performance. Your users—and regulators—will thank you.