In the previous episodes of this series, we explored interactive dashboards with Plotly and Streamlit, then tackled the pitfalls of spurious correlations. Now we face one of the most pervasive challenges in real-world data science: class imbalance. When 99.8% of your credit card transactions are legitimate and only 0.2% are fraudulent, standard machine learning approaches crumble. This episode dives deep into the techniques that fix this — from synthetic oversampling to cost-sensitive learning — with hands-on code using a real Kaggle dataset.
Why Accuracy Is a Lie
Imagine building a fraud detection model that achieves 99.8% accuracy. Sounds impressive, right? But if you simply predict “not fraud” for every single transaction, you get that exact number. The model catches zero fraud — it’s completely useless.
This is the accuracy paradox. When classes are heavily skewed, accuracy becomes meaningless. We need metrics that tell us what actually matters.
Metrics That Matter for Imbalanced Data
| Metric | Formula | What It Tells You |
|---|---|---|
| Precision | Of predicted positives, how many are correct? | |
| Recall | Of actual positives, how many did we catch? | |
| F1-Score | Harmonic mean of precision and recall | |
| PR-AUC | Area under Precision-Recall curve | Overall ranking quality for the minority class |
| ROC-AUC | Area under ROC curve | Discrimination ability across all thresholds |
Where = true positives, = false positives, = false negatives.
Key insight: For imbalanced datasets, PR-AUC is more informative than ROC-AUC. ROC-AUC can appear optimistically high because the true negative rate stays inflated by the majority class. PR-AUC focuses exclusively on how well you handle the minority class.
Setting Up: Loading the Credit Card Fraud Dataset
We’ll use the famous Kaggle Credit Card Fraud Detection dataset throughout this episode. It contains 284,807 transactions with only 492 frauds (0.173%).
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
classification_report, f1_score,
precision_recall_curve, auc, roc_auc_score
)
import warnings
warnings.filterwarnings('ignore')
# Load dataset
df = pd.read_csv('creditcard.csv')
print(f"Total transactions: {len(df):,}")
print(f"Fraudulent: {df['Class'].sum():,} ({df['Class'].mean()*100:.3f}%)")
print(f"Legitimate: {(df['Class']==0).sum():,} ({(1-df['Class'].mean())*100:.3f}%)")
# Prepare features and target
X = df.drop('Class', axis=1)
y = df['Class']
# Scale the 'Amount' and 'Time' columns (V1-V28 are already PCA-transformed)
scaler = StandardScaler()
X['Amount'] = scaler.fit_transform(X[['Amount']])
X['Time'] = scaler.fit_transform(X[['Time']])
# Train-test split with stratification to preserve class ratio
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=42, stratify=y
)
print(f"\nTraining set fraud ratio: {y_train.mean()*100:.3f}%")
print(f"Test set fraud ratio: {y_test.mean()*100:.3f}%")
Oversampling: Generating Synthetic Minority Samples
SMOTE: The Foundation
Synthetic Minority Over-sampling Technique (SMOTE) is the most widely used approach. Instead of simply duplicating minority samples (which causes overfitting), SMOTE creates new synthetic samples by interpolating between existing minority instances.
The algorithm works as follows:
- For each minority sample , find its nearest minority neighbors
- Randomly select one neighbor
- Generate a synthetic sample along the line segment between them:
Where is a random number drawn from a uniform distribution between 0 and 1. This places the new point somewhere on the line connecting the original sample to its neighbor.
from imblearn.over_sampling import SMOTE, BorderlineSMOTE, ADASYN
# Standard SMOTE
smote = SMOTE(random_state=42, k_neighbors=5)
X_smote, y_smote = smote.fit_resample(X_train, y_train)
print(f"Before SMOTE: {dict(pd.Series(y_train).value_counts())}")
print(f"After SMOTE: {dict(pd.Series(y_smote).value_counts())}")
Borderline-SMOTE: Smarter Sample Selection
Standard SMOTE treats all minority samples equally, but not all are equally important. Borderline-SMOTE focuses synthesis on minority samples that lie near the decision boundary — the ones that actually matter for classification.
It classifies each minority sample into three categories:
– Safe: most neighbors are minority → easy to classify, less useful
– Danger: roughly half neighbors are majority → on the boundary, most useful
– Noise: most neighbors are majority → likely an outlier
Only danger samples are used as seeds for synthesis.
# Borderline-SMOTE
bsmote = BorderlineSMOTE(random_state=42, kind='borderline-1')
X_bsmote, y_bsmote = bsmote.fit_resample(X_train, y_train)
print(f"After Borderline-SMOTE: {dict(pd.Series(y_bsmote).value_counts())}")
ADASYN: Adaptive Synthetic Sampling
ADASYN (Adaptive Synthetic Sampling) takes it further by generating more synthetic samples for minority instances that are harder to learn. The density of generated samples is proportional to the local difficulty.
For each minority sample , ADASYN computes a density ratio:
Where is the number of majority-class samples among the nearest neighbors of , and is a normalization constant. The number of synthetic samples generated from is then:
Where is the total number of synthetic samples to generate. This means minority samples surrounded by more majority samples (harder cases) get more synthetic neighbors.
# ADASYN
adasyn = ADASYN(random_state=42, n_neighbors=5)
X_adasyn, y_adasyn = adasyn.fit_resample(X_train, y_train)
print(f"After ADASYN: {dict(pd.Series(y_adasyn).value_counts())}")
# Note: ADASYN may not produce exact 50/50 balance — this is by design
Undersampling: Reducing the Majority Class
Instead of creating more minority samples, we can reduce majority samples. This is especially useful when data is abundant and training time is a concern.
Random Undersampling
The simplest approach — randomly remove majority samples until classes are balanced. Fast but risks losing important information.
from imblearn.under_sampling import (
RandomUnderSampler, TomekLinks, NearMiss
)
# Random undersampling
rus = RandomUnderSampler(random_state=42)
X_rus, y_rus = rus.fit_resample(X_train, y_train)
print(f"After RandomUnderSampler: {dict(pd.Series(y_rus).value_counts())}")
Tomek Links: Cleaning the Boundary
A Tomek link is a pair of samples from different classes where each is the other’s nearest neighbor. These are ambiguous, boundary-straddling pairs. Removing the majority-class member of each Tomek link cleans the decision boundary.
# Tomek Links
tomek = TomekLinks()
X_tomek, y_tomek = tomek.fit_resample(X_train, y_train)
print(f"After Tomek Links: {dict(pd.Series(y_tomek).value_counts())}")
# Tomek Links removes relatively few samples — it's a cleaning technique
NearMiss: Heuristic Undersampling
NearMiss selects majority samples that are closest to minority samples, keeping only the most informative majority instances. Version 1 keeps majority samples whose average distance to the closest minority samples is smallest.
# NearMiss (version 1)
nm = NearMiss(version=1, n_neighbors=3)
X_nm, y_nm = nm.fit_resample(X_train, y_train)
print(f"After NearMiss: {dict(pd.Series(y_nm).value_counts())}")
Combined Approaches: Best of Both Worlds
The most effective strategies often combine oversampling and undersampling.
SMOTEENN
SMOTEENN first applies SMOTE to oversample the minority class, then uses Edited Nearest Neighbors (ENN) to clean up. ENN removes any sample whose class label differs from the majority of its nearest neighbors — effectively removing noisy synthetic samples and borderline majority samples.
from imblearn.combine import SMOTEENN, SMOTETomek
# SMOTEENN
smote_enn = SMOTEENN(random_state=42)
X_senn, y_senn = smote_enn.fit_resample(X_train, y_train)
print(f"After SMOTEENN: {dict(pd.Series(y_senn).value_counts())}")
SMOTETomek
SMOTETomek applies SMOTE followed by Tomek Links removal. Less aggressive than SMOTEENN, it only removes the ambiguous Tomek pairs.
# SMOTETomek
smote_tomek = SMOTETomek(random_state=42)
X_stomek, y_stomek = smote_tomek.fit_resample(X_train, y_train)
print(f"After SMOTETomek: {dict(pd.Series(y_stomek).value_counts())}")
Cost-Sensitive Learning: No Resampling Needed
Instead of modifying the data, we can modify the learning algorithm to pay more attention to minority samples.
class_weight in scikit-learn
Most scikit-learn classifiers support class_weight='balanced', which automatically adjusts weights inversely proportional to class frequencies:
Where is the total number of samples, is the number of classes, and is the number of samples in class . For our fraud dataset with 0.173% fraud, the fraud class gets a weight roughly 289 times higher than legitimate transactions.
# Cost-sensitive logistic regression
lr_weighted = LogisticRegression(
class_weight='balanced',
max_iter=1000,
random_state=42
)
lr_weighted.fit(X_train, y_train)
y_pred_weighted = lr_weighted.predict(X_test)
print(classification_report(y_test, y_pred_weighted, digits=4))
Focal Loss: Down-Weighting Easy Examples
Originally proposed for object detection (Lin et al., 2017), focal loss modifies cross-entropy by adding a modulating factor:
Where is the predicted probability for the true class, is a class balancing factor, and is the focusing parameter (typically 2). When a sample is well-classified ( is high), the factor becomes very small, reducing its contribution to the loss. Misclassified samples retain high loss values.
import tensorflow as tf
def focal_loss(gamma=2.0, alpha=0.25):
"""Focal loss for binary classification."""
def loss_fn(y_true, y_pred):
y_true = tf.cast(y_true, tf.float32)
# Clip predictions to prevent log(0)
y_pred = tf.clip_by_value(y_pred, 1e-7, 1 - 1e-7)
# Compute focal loss components
p_t = y_true * y_pred + (1 - y_true) * (1 - y_pred)
alpha_t = y_true * alpha + (1 - y_true) * (1 - alpha)
focal_weight = alpha_t * tf.pow(1 - p_t, gamma)
# Binary cross-entropy weighted by focal factor
bce = -(y_true * tf.math.log(y_pred) +
(1 - y_true) * tf.math.log(1 - y_pred))
return tf.reduce_mean(focal_weight * bce)
return loss_fn
Comprehensive Benchmark: All Methods Head-to-Head
Let’s run every technique through a standardized pipeline and compare results on the Credit Card Fraud dataset.
import time
from sklearn.ensemble import RandomForestClassifier
def evaluate_method(X_tr, y_tr, X_te, y_te, method_name):
"""Train a Random Forest and return evaluation metrics."""
start = time.time()
clf = RandomForestClassifier(
n_estimators=100, random_state=42, n_jobs=-1
)
clf.fit(X_tr, y_tr)
train_time = time.time() - start
y_pred = clf.predict(X_te)
y_prob = clf.predict_proba(X_te)[:, 1]
precision_vals, recall_vals, _ = precision_recall_curve(y_te, y_prob)
pr_auc_val = auc(recall_vals, precision_vals)
report = classification_report(y_te, y_pred, output_dict=True)
return {
'Method': method_name,
'Precision': report['1']['precision'],
'Recall': report['1']['recall'],
'F1': report['1']['f1-score'],
'PR-AUC': pr_auc_val,
'ROC-AUC': roc_auc_score(y_te, y_prob),
'Train Time (s)': round(train_time, 2),
'Train Size': len(X_tr)
}
# Define all resampled datasets
datasets = {
'Baseline (No Resampling)': (X_train, y_train),
'SMOTE': (X_smote, y_smote),
'Borderline-SMOTE': (X_bsmote, y_bsmote),
'ADASYN': (X_adasyn, y_adasyn),
'RandomUnderSampler': (X_rus, y_rus),
'Tomek Links': (X_tomek, y_tomek),
'NearMiss': (X_nm, y_nm),
'SMOTEENN': (X_senn, y_senn),
'SMOTETomek': (X_stomek, y_stomek),
}
# Add cost-sensitive learning (uses original data with class_weight)
start = time.time()
clf_cw = RandomForestClassifier(
n_estimators=100, random_state=42, n_jobs=-1,
class_weight='balanced'
)
clf_cw.fit(X_train, y_train)
cw_time = time.time() - start
y_pred_cw = clf_cw.predict(X_test)
y_prob_cw = clf_cw.predict_proba(X_test)[:, 1]
prec_cw, rec_cw, _ = precision_recall_curve(y_test, y_prob_cw)
report_cw = classification_report(y_test, y_pred_cw, output_dict=True)
# Evaluate all resampling methods
results = []
for name, (X_r, y_r) in datasets.items():
results.append(evaluate_method(X_r, y_r, X_test, y_test, name))
# Append cost-sensitive result
results.append({
'Method': 'Class Weight (balanced)',
'Precision': report_cw['1']['precision'],
'Recall': report_cw['1']['recall'],
'F1': report_cw['1']['f1-score'],
'PR-AUC': auc(rec_cw, prec_cw),
'ROC-AUC': roc_auc_score(y_test, y_prob_cw),
'Train Time (s)': round(cw_time, 2),
'Train Size': len(X_train)
})
results_df = pd.DataFrame(results)
print(results_df.to_string(index=False))
Benchmark Results
Typical results on this dataset (Random Forest, 100 trees):
| Method | Precision | Recall | F1 | PR-AUC | ROC-AUC | Train Time (s) | Train Size |
|---|---|---|---|---|---|---|---|
| Baseline (No Resampling) | 0.9630 | 0.7551 | 0.8462 | 0.8497 | 0.9526 | 8.2 | 199,364 |
| SMOTE | 0.8721 | 0.8367 | 0.8540 | 0.8601 | 0.9584 | 17.5 | 398,040 |
| Borderline-SMOTE | 0.8955 | 0.8163 | 0.8541 | 0.8582 | 0.9571 | 17.1 | 398,040 |
| ADASYN | 0.8502 | 0.8435 | 0.8468 | 0.8553 | 0.9562 | 17.8 | 398,126 |
| RandomUnderSampler | 0.0422 | 0.9184 | 0.0807 | 0.7431 | 0.9378 | 0.1 | 688 |
| Tomek Links | 0.9619 | 0.7551 | 0.8457 | 0.8493 | 0.9524 | 8.1 | 199,256 |
| NearMiss | 0.0061 | 0.9796 | 0.0121 | 0.4823 | 0.8652 | 0.1 | 688 |
| SMOTEENN | 0.7124 | 0.8571 | 0.7782 | 0.8419 | 0.9501 | 14.3 | 354,892 |
| SMOTETomek | 0.8698 | 0.8367 | 0.8529 | 0.8598 | 0.9581 | 17.4 | 397,838 |
| Class Weight (balanced) | 0.8512 | 0.8571 | 0.8542 | 0.8602 | 0.9587 | 8.3 | 199,364 |
Key takeaways: SMOTE and its variants meaningfully improve recall without catastrophic precision loss. Pure undersampling (NearMiss, RandomUnderSampler) destroys precision because too much majority-class information is lost. Class weight balancing achieves competitive F1 with zero resampling overhead — often the best starting point.
Interactive Visualization: Decision Boundaries with Plotly
Let’s visualize how resampling changes the decision boundary. We’ll use two PCA components for a 2D view.
import plotly.graph_objects as go
from plotly.subplots import make_subplots
from sklearn.decomposition import PCA
# Reduce to 2D for visualization
pca = PCA(n_components=2, random_state=42)
X_train_2d = pca.fit_transform(X_train)
X_test_2d = pca.transform(X_test)
# Also transform SMOTE-resampled data
X_smote_2d = pca.transform(X_smote)
def plot_decision_boundary(X_2d, y, title, clf=None, X_test_2d=None, y_test=None):
"""Create an interactive Plotly decision boundary plot."""
# Create mesh grid for decision boundary
x_min, x_max = X_2d[:, 0].min() - 1, X_2d[:, 0].max() + 1
y_min, y_max = X_2d[:, 1].min() - 1, X_2d[:, 1].max() + 1
xx, yy = np.meshgrid(
np.linspace(x_min, x_max, 200),
np.linspace(y_min, y_max, 200)
)
if clf is not None:
Z = clf.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:, 1]
Z = Z.reshape(xx.shape)
fig = go.Figure()
# Decision boundary as contour
if clf is not None:
fig.add_trace(go.Contour(
x=np.linspace(x_min, x_max, 200),
y=np.linspace(y_min, y_max, 200),
z=Z,
colorscale='RdBu_r',
opacity=0.4,
showscale=True,
colorbar=dict(title='P(Fraud)'),
contours=dict(showlines=False)
))
# Plot test points
if X_test_2d is not None and y_test is not None:
for cls, name, color, symbol in [
(0, 'Legitimate', '#2196F3', 'circle'),
(1, 'Fraud', '#F44336', 'diamond')
]:
mask = y_test == cls
fig.add_trace(go.Scatter(
x=X_test_2d[mask, 0],
y=X_test_2d[mask, 1],
mode='markers',
name=name,
marker=dict(
color=color, size=5 if cls == 0 else 8,
opacity=0.5 if cls == 0 else 0.9,
symbol=symbol,
line=dict(width=0.5, color='white')
),
hovertemplate=f'{name}<br>PC1: %{{x:.2f}}<br>PC2: %{{y:.2f}}'
))
fig.update_layout(
title=dict(text=title, font=dict(size=16)),
xaxis_title='Principal Component 1',
yaxis_title='Principal Component 2',
width=700, height=500,
template='plotly_white'
)
return fig
# Train classifiers on 2D data for visualization
from sklearn.ensemble import GradientBoostingClassifier
# Baseline model (no resampling)
clf_baseline = GradientBoostingClassifier(
n_estimators=100, random_state=42, max_depth=3
)
clf_baseline.fit(X_train_2d, y_train)
# SMOTE model
clf_smote = GradientBoostingClassifier(
n_estimators=100, random_state=42, max_depth=3
)
smote_2d = SMOTE(random_state=42)
X_train_2d_smote, y_train_smote = smote_2d.fit_resample(X_train_2d, y_train)
clf_smote.fit(X_train_2d_smote, y_train_smote)
# Generate comparison plots
fig1 = plot_decision_boundary(
X_train_2d, y_train,
'Decision Boundary: No Resampling',
clf_baseline, X_test_2d, y_test
)
fig1.show()
fig2 = plot_decision_boundary(
X_train_2d_smote, y_train_smote,
'Decision Boundary: After SMOTE',
clf_smote, X_test_2d, y_test
)
fig2.show()
The SMOTE-resampled boundary typically shows a wider and more confident fraud detection region, though it may also expand into legitimate territory — the classic precision-recall tradeoff made visible.
Interactive PR Curve Comparison
def plot_pr_curves(results_dict, y_test):
"""Plot interactive Precision-Recall curves for all methods."""
fig = go.Figure()
colors = [
'#1f77b4', '#ff7f0e', '#2ca02c', '#d62728',
'#9467bd', '#8c564b', '#e377c2', '#7f7f7f',
'#bcbd22', '#17becf'
]
for i, (name, y_prob) in enumerate(results_dict.items()):
precision_vals, recall_vals, thresholds = precision_recall_curve(
y_test, y_prob
)
pr_auc_val = auc(recall_vals, precision_vals)
fig.add_trace(go.Scatter(
x=recall_vals,
y=precision_vals,
mode='lines',
name=f'{name} (AUC={pr_auc_val:.4f})',
line=dict(color=colors[i % len(colors)], width=2),
hovertemplate=(
f'{name}<br>'
'Recall: %{x:.3f}<br>'
'Precision: %{y:.3f}<extra></extra>'
)
))
fig.update_layout(
title='Precision-Recall Curves: All Methods Compared',
xaxis_title='Recall',
yaxis_title='Precision',
width=800, height=550,
template='plotly_white',
legend=dict(x=0.02, y=0.02, bgcolor='rgba(255,255,255,0.8)'),
xaxis=dict(range=[0, 1.02]),
yaxis=dict(range=[0, 1.02])
)
return fig
# Collect probability predictions from each method
prob_dict = {}
for name, (X_r, y_r) in datasets.items():
clf = RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1)
clf.fit(X_r, y_r)
prob_dict[name] = clf.predict_proba(X_test)[:, 1]
prob_dict['Class Weight (balanced)'] = y_prob_cw
fig_pr = plot_pr_curves(prob_dict, y_test)
fig_pr.show()
This interactive plot lets you hover over each curve to see the exact precision-recall tradeoff at any threshold — essential for choosing the right operating point for your specific use case.
Practical Guide: The imbalanced-learn Pipeline
The imbalanced-learn library integrates seamlessly with scikit-learn through its Pipeline class. This is the recommended way to use resampling in production.
from imblearn.pipeline import Pipeline as ImbPipeline
from sklearn.model_selection import cross_val_score, StratifiedKFold
# Build a proper pipeline (resampling happens only on training folds)
pipeline = ImbPipeline([
('scaler', StandardScaler()),
('sampler', SMOTE(random_state=42)),
('classifier', RandomForestClassifier(
n_estimators=100, random_state=42, n_jobs=-1
))
])
# Cross-validation with stratified folds
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(
pipeline, X_train, y_train,
cv=cv, scoring='f1', n_jobs=-1
)
print(f"Cross-validated F1 scores: {scores}")
print(f"Mean F1: {scores.mean():.4f} (+/- {scores.std():.4f})")
Critical warning: Never apply resampling before splitting your data. Resampling must happen inside cross-validation folds (only on training data). Otherwise, synthetic samples derived from test data leak information into training, giving you falsely optimistic results. The
imblearn.pipeline.Pipelinehandles this correctly.
Hyperparameter Tuning with Resampling
from sklearn.model_selection import GridSearchCV
# Define parameter grid (including sampler params)
param_grid = {
'sampler__k_neighbors': [3, 5, 7],
'classifier__n_estimators': [100, 200],
'classifier__max_depth': [None, 10, 20],
}
grid_search = GridSearchCV(
pipeline,
param_grid,
cv=StratifiedKFold(n_splits=3, shuffle=True, random_state=42),
scoring='f1',
n_jobs=-1,
verbose=1
)
grid_search.fit(X_train, y_train)
print(f"Best parameters: {grid_search.best_params_}")
print(f"Best F1: {grid_search.best_score_:.4f}")
# Evaluate on test set
y_pred_best = grid_search.predict(X_test)
print(classification_report(y_test, y_pred_best, digits=4))
Choosing the Right Strategy: A Decision Framework
With so many options, here’s a practical guide:
| Scenario | Recommended Approach |
|---|---|
| Quick baseline | class_weight='balanced' — zero resampling overhead |
| Moderate imbalance (1:10 to 1:100) | SMOTE or Borderline-SMOTE |
| Severe imbalance (1:1000+) | SMOTEENN or ADASYN + cost-sensitive learning |
| Very large dataset | Undersampling (Random or Tomek Links) to reduce training time |
| Noisy dataset | Borderline-SMOTE or SMOTEENN (built-in noise handling) |
| Production pipeline | imblearn.Pipeline with cross-validated tuning |
Rule of thumb: Start with
class_weight='balanced'. If that’s insufficient, try SMOTE. Only reach for more complex methods when simpler ones demonstrably fail on your validation set.
Conclusion
Class imbalance isn’t just a nuisance — it’s a fundamental challenge that can make or break your model in production. In this episode, we moved beyond naive accuracy to metrics that truly matter (F1, PR-AUC), explored the mathematical foundations of SMOTE and ADASYN, and benchmarked ten different strategies on real fraud detection data.
The key lessons: accuracy is deceptive for imbalanced problems, SMOTE-family methods provide a reliable boost in recall without devastating precision, and cost-sensitive learning via class_weight is an underappreciated first line of defense that requires no data modification at all. Always resample inside your cross-validation loop, and let the PR curve — not a single threshold metric — guide your final model selection.
In the final episode of this series, we’ll shift from what the model predicts to why it predicts, exploring Explainable AI with SHAP and LIME to open the black box of complex models.
Did you find this helpful?
☕ Buy me a coffee
Leave a Reply