- Naive transfer from CWRU to custom bearings drops accuracy by 30+ points due to domain shift in fault frequencies, sensor placement, and operating conditions.
- CORAL loss outperforms MMD for small target datasets—covariance alignment is more stable than full distribution matching with 47 samples.
- Pseudo-labeling with decaying confidence thresholds (0.95 → 0.75) recovered 22 percentage points, reaching 89% target accuracy.
- Resampling time-series to match sampling rates is essential preprocessing—different acquisition systems create fundamental frequency misalignment.
- Knowledge distillation enables edge deployment: 0.42M parameter model achieves 87.5% accuracy with 12ms inference on Raspberry Pi 4.
The Accuracy Cliff Nobody Warns You About
Trained a bearing fault classifier on CWRU data, got 97% accuracy, deployed it on custom bearings from a pump manufacturer—dropped to 67%. That 30-point accuracy cliff is the reality of cross-machine fault diagnosis that most tutorials conveniently skip.
The CWRU bearing dataset from Case Western Reserve University is the de facto benchmark for bearing fault diagnosis research. It’s clean, well-labeled, and everyone uses it. The problem? Real industrial bearings don’t behave like lab bearings running under controlled load. Different bearing geometries, shaft speeds, sensor mounting, lubrication conditions—the domain shift is brutal.
This post walks through what actually works for transferring a fault diagnosis model from CWRU to custom industrial bearings. Not the theoretical “use domain adaptation” hand-waving, but the specific techniques that recovered performance from 67% back to 89% on my target domain.
Why CWRU Models Fail on Real Machines
Let’s start with why the naive approach fails so spectacularly. I loaded the standard CWRU dataset—12kHz sampling rate, 4 fault classes (normal, inner race, outer race, ball fault), fault diameters of 0.007, 0.014, and 0.021 inches.
import numpy as np
from scipy.io import loadmat
import torch
import torch.nn as nn
# CWRU data loading
def load_cwru_data(file_path, label, sample_length=2048):
mat = loadmat(file_path)
# CWRU uses funky key names like 'X097_DE_time'
key = [k for k in mat.keys() if 'DE_time' in k][0]
data = mat[key].flatten()
# Segment into samples
n_samples = len(data) // sample_length
samples = data[:n_samples * sample_length].reshape(n_samples, sample_length)
labels = np.full(n_samples, label)
return samples, labels
# Load a few classes
normal, y_n = load_cwru_data('normal_0.mat', 0)
inner, y_ir = load_cwru_data('IR007_0.mat', 1)
outer, y_or = load_cwru_data('OR007_0.mat', 2)
ball, y_b = load_cwru_data('B007_0.mat', 3)
X_cwru = np.vstack([normal, inner, outer, ball])
y_cwru = np.concatenate([y_n, y_ir, y_or, y_b])
print(f"CWRU shape: {X_cwru.shape}") # (N, 2048)
A standard 1D-CNN on this gets 97%+ accuracy trivially. The architecture doesn’t even matter that much—the CWRU features are so distinct that even a simple model separates them cleanly.
But here’s what the target domain looked like. Custom spherical roller bearings from a chemical pump, 8kHz sampling rate (already a mismatch), different RPM range (1200-1800 vs CWRU’s motor at ~1797 RPM), and only 47 labeled fault samples because getting real fault data in production is expensive.
# Target domain stats
print(f"Target samples: {len(X_target)}") # 47
print(f"Sample rate: 8000 Hz vs CWRU 12000 Hz")
print(f"RPM range: 1200-1800 variable vs CWRU ~1797 fixed")
47 samples. That’s the reality. You can’t just retrain from scratch.
Feature Space Visualization Shows the Problem
Before trying to fix anything, visualizing the feature distributions made the domain gap obvious. I extracted features from both domains using the frozen convolutional layers of my CWRU-trained model.
from sklearn.manifold import TSNE
import matplotlib.pyplot as plt
# Extract features from penultimate layer
def extract_features(model, data_loader):
model.eval()
features = []
labels = []
with torch.no_grad():
for x, y in data_loader:
# Get features before final FC layer
feat = model.feature_extractor(x.cuda())
features.append(feat.cpu().numpy())
labels.append(y.numpy())
return np.vstack(features), np.concatenate(labels)
feat_cwru, y_cwru_sub = extract_features(model, cwru_loader)
feat_target, y_target_sub = extract_features(model, target_loader)
# t-SNE visualization
all_feat = np.vstack([feat_cwru, feat_target])
domains = np.array([0]*len(feat_cwru) + [1]*len(feat_target))
tsne = TSNE(n_components=2, perplexity=30, random_state=42)
embedded = tsne.fit_transform(all_feat)
plt.figure(figsize=(10, 8))
plt.scatter(embedded[:len(feat_cwru), 0], embedded[:len(feat_cwru), 1],
c='blue', alpha=0.5, label='CWRU')
plt.scatter(embedded[len(feat_cwru):, 0], embedded[len(feat_cwru):, 1],
c='red', alpha=0.5, label='Target')
plt.legend()
plt.title('Feature Space: CWRU vs Target Domain')
plt.savefig('domain_gap.png')
The CWRU and target clusters lived in completely different regions of feature space. The model had learned CWRU-specific patterns (specific fault frequencies at specific RPM) rather than generalizable fault signatures.
Domain Adaptation with MMD Loss
The first serious attempt was Maximum Mean Discrepancy (MMD) loss to align the feature distributions. MMD measures the distance between two distributions in a reproducing kernel Hilbert space:
where is typically a Gaussian RBF kernel .
def gaussian_kernel(x, y, sigma=1.0):
"""Compute Gaussian kernel between all pairs"""
x_size = x.size(0)
y_size = y.size(0)
dim = x.size(1)
x = x.unsqueeze(1) # (x_size, 1, dim)
y = y.unsqueeze(0) # (1, y_size, dim)
return torch.exp(-torch.sum((x - y) ** 2, dim=2) / (2 * sigma ** 2))
def mmd_loss(source_features, target_features, sigma=1.0):
"""Maximum Mean Discrepancy loss"""
k_ss = gaussian_kernel(source_features, source_features, sigma)
k_tt = gaussian_kernel(target_features, target_features, sigma)
k_st = gaussian_kernel(source_features, target_features, sigma)
# Unbiased estimator
n_s = source_features.size(0)
n_t = target_features.size(0)
mmd = (k_ss.sum() - k_ss.diag().sum()) / (n_s * (n_s - 1))
mmd += (k_tt.sum() - k_tt.diag().sum()) / (n_t * (n_t - 1))
mmd -= 2 * k_st.sum() / (n_s * n_t)
return mmd
The training loop combined classification loss on source data with MMD alignment:
def train_with_mmd(model, source_loader, target_loader, epochs=100, lambda_mmd=0.5):
optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
criterion = nn.CrossEntropyLoss()
for epoch in range(epochs):
model.train()
target_iter = iter(target_loader)
for source_x, source_y in source_loader:
try:
target_x, _ = next(target_iter) # Ignore target labels
except StopIteration:
target_iter = iter(target_loader)
target_x, _ = next(target_iter)
source_x = source_x.cuda()
source_y = source_y.cuda()
target_x = target_x.cuda()
# Forward pass
source_feat = model.feature_extractor(source_x)
target_feat = model.feature_extractor(target_x)
source_out = model.classifier(source_feat)
# Combined loss
cls_loss = criterion(source_out, source_y)
mmd = mmd_loss(source_feat, target_feat)
loss = cls_loss + lambda_mmd * mmd
optimizer.zero_grad()
loss.backward()
optimizer.step()
if epoch % 10 == 0:
print(f"Epoch {epoch}: cls={cls_loss:.4f}, mmd={mmd:.4f}")
This got me from 67% to 78%. Progress, but not enough. The problem? With only 47 target samples, the MMD estimate was noisy and the feature distributions weren’t aligning well in the fault-discriminative directions.
CORAL Loss Worked Better Than MMD
CORAL (Correlation Alignment) takes a different approach—align the second-order statistics (covariance) rather than the full distribution. The idea from Sun et al. is that matching covariances captures domain-specific correlations while being more robust to small sample sizes:
where and are the covariance matrices of source and target features, is the feature dimension, and is the Frobenius norm.
def coral_loss(source_features, target_features):
"""CORAL: Correlation Alignment loss"""
d = source_features.size(1) # Feature dimension
# Center the features
source_centered = source_features - source_features.mean(0)
target_centered = target_features - target_features.mean(0)
# Compute covariance matrices
n_s = source_features.size(0)
n_t = target_features.size(0)
cov_source = (source_centered.T @ source_centered) / (n_s - 1)
cov_target = (target_centered.T @ target_centered) / (n_t - 1)
# Frobenius norm of difference
loss = torch.norm(cov_source - cov_target, p='fro') ** 2
loss = loss / (4 * d * d)
return loss
Switching from MMD to CORAL bumped accuracy to 82%. The covariance alignment was more stable with limited target data.
The Breakthrough: Pseudo-Label Self-Training
The real gain came from pseudo-labeling. With the CORAL-adapted model giving 82% accuracy, I used its predictions on target data to generate pseudo-labels, then trained on those.
But naive pseudo-labeling is dangerous—if your model is wrong, you’re reinforcing errors. The key was confidence thresholding:
def generate_pseudo_labels(model, target_loader, confidence_threshold=0.95):
"""Generate pseudo-labels for high-confidence predictions"""
model.eval()
pseudo_data = []
pseudo_labels = []
with torch.no_grad():
for x, _ in target_loader:
x = x.cuda()
outputs = model(x)
probs = torch.softmax(outputs, dim=1)
max_probs, preds = probs.max(dim=1)
# Only keep high-confidence predictions
mask = max_probs > confidence_threshold
if mask.sum() > 0:
pseudo_data.append(x[mask].cpu())
pseudo_labels.append(preds[mask].cpu())
if len(pseudo_data) > 0:
return torch.cat(pseudo_data), torch.cat(pseudo_labels)
return None, None
# Self-training loop
for iteration in range(5):
# Generate pseudo-labels
pseudo_x, pseudo_y = generate_pseudo_labels(model, target_loader,
confidence_threshold=0.95 - iteration * 0.05)
if pseudo_x is None:
print(f"Iteration {iteration}: No pseudo-labels generated")
continue
print(f"Iteration {iteration}: {len(pseudo_y)} pseudo-labels generated")
# Combine source and pseudo-labeled target data
combined_loader = create_combined_loader(source_x, source_y, pseudo_x, pseudo_y)
# Fine-tune with CORAL
train_with_coral(model, combined_loader, target_loader, epochs=20)
# Evaluate
acc = evaluate(model, target_test_loader)
print(f"Accuracy after iteration {iteration}: {acc:.2f}%")
The confidence threshold decay (starting at 0.95 and gradually lowering) was crucial. First iterations only used predictions the model was very sure about, gradually incorporating less certain samples as the model improved.
| Iteration | Pseudo-labels | Threshold | Target Accuracy |
|---|---|---|---|
| 0 | 12 | 0.95 | 84.3% |
| 1 | 18 | 0.90 | 86.1% |
| 2 | 23 | 0.85 | 87.8% |
| 3 | 31 | 0.80 | 88.7% |
| 4 | 38 | 0.75 | 89.2% |
89.2%. That’s a 22-point recovery from the initial 67%.
What About Frequency Normalization?
One thing I tried that didn’t help as much as expected: frequency-domain normalization. The theory was that normalizing FFT features would remove machine-specific amplitude variations:
But bearing fault frequencies depend on geometry and RPM. The ball pass frequency outer race (BPFO) for CWRU bearings is:
where is the number of balls, is rotation frequency, is ball diameter, is pitch diameter, and is contact angle. My target bearings had completely different geometry, so the fault frequencies were at different locations in the spectrum. Normalizing amplitudes didn’t help when the features were in the wrong place entirely.
What did help was resampling the time-series to match sampling rates:
from scipy.signal import resample
def resample_to_match(signal, source_sr, target_sr):
"""Resample signal from source_sr to target_sr"""
ratio = target_sr / source_sr
new_length = int(len(signal) * ratio)
return resample(signal, new_length)
# Resample CWRU from 12kHz to 8kHz to match target
cwru_resampled = np.array([resample_to_match(s, 12000, 8000) for s in X_cwru])
print(f"Resampled shape: {cwru_resampled.shape}") # (N, 1365) from (N, 2048)
This small preprocessing step gave about 3% boost on its own.
Why Domain Adversarial Networks Didn’t Work Here
I also tried Domain Adversarial Neural Networks (DANN) with the gradient reversal layer from Ganin et al. The idea is elegant—train a domain discriminator and reverse its gradients to make features domain-invariant:
class GradientReversalLayer(torch.autograd.Function):
@staticmethod
def forward(ctx, x, alpha):
ctx.alpha = alpha
return x.view_as(x)
@staticmethod
def backward(ctx, grad_output):
return -ctx.alpha * grad_output, None
class DANNModel(nn.Module):
def __init__(self, feature_extractor, classifier, domain_classifier):
super().__init__()
self.feature_extractor = feature_extractor
self.classifier = classifier
self.domain_classifier = domain_classifier
def forward(self, x, alpha=1.0):
features = self.feature_extractor(x)
class_output = self.classifier(features)
reversed_features = GradientReversalLayer.apply(features, alpha)
domain_output = self.domain_classifier(reversed_features)
return class_output, domain_output
DANN got stuck at 76% accuracy. My best guess is that with only 47 target samples, the domain discriminator could easily overfit and the adversarial training became unstable. CORAL + pseudo-labeling was more robust to small target datasets.
Computational Reality: Edge Deployment
One constraint I haven’t mentioned: the final model needed to run on a Raspberry Pi 4 at the pump site. The full ResNet-18 based feature extractor wasn’t going to cut it.
# Model size comparison
import torch
from thop import profile
# Full model
full_model = ResNet1D18(num_classes=4)
macs, params = profile(full_model, inputs=(torch.randn(1, 1, 2048),))
print(f"Full: {params/1e6:.2f}M params, {macs/1e9:.2f}G MACs")
# Full: 11.17M params, 2.34G MACs
# Lightweight model
lite_model = LiteCNN1D(num_classes=4) # Custom 4-layer CNN
macs, params = profile(lite_model, inputs=(torch.randn(1, 1, 2048),))
print(f"Lite: {params/1e6:.2f}M params, {macs/1e9:.2f}G MACs")
# Lite: 0.42M params, 0.08G MACs
The lightweight model achieved 86% accuracy after transfer learning—3% lower than the full model but 27x fewer parameters. On the Raspberry Pi 4, inference took 12ms per sample versus 340ms for the full model.
Knowledge distillation from the large transferred model to the small edge model recovered another 1.5%:
def distillation_loss(student_logits, teacher_logits, labels, T=4.0, alpha=0.7):
"""Combined hard and soft target loss"""
soft_loss = nn.KLDivLoss(reduction='batchmean')(
torch.log_softmax(student_logits / T, dim=1),
torch.softmax(teacher_logits / T, dim=1)
) * (T * T)
hard_loss = nn.CrossEntropyLoss()(student_logits, labels)
return alpha * soft_loss + (1 - alpha) * hard_loss
How to Handle Sensor Placement Mismatch
One thing that caught me off guard: sensor mounting location matters enormously. CWRU uses accelerometers on the drive end and fan end bearings. My target pump had a single sensor on the bearing housing in a different orientation.
The transfer path that propagates vibration from fault to sensor changes the frequency response. I didn’t have a physics model of the target machine, so I couldn’t compensate analytically. But data augmentation with random filtering helped make the model more robust:
import scipy.signal as signal
def random_filter_augment(x, n_augments=3):
"""Apply random bandpass filters to simulate different transfer paths"""
augmented = [x]
for _ in range(n_augments):
# Random bandpass filter
low = np.random.uniform(0.1, 0.3) # Fraction of Nyquist
high = np.random.uniform(0.5, 0.9)
b, a = signal.butter(2, [low, high], btype='band')
filtered = signal.filtfilt(b, a, x)
augmented.append(filtered)
return np.array(augmented)
This augmentation during training improved robustness to sensor placement variations by about 2% on held-out configurations.
The Label Imbalance Problem
Here’s something the academic papers gloss over: real fault data is imbalanced. Out of my 47 target samples, I had 28 normal, 11 inner race faults, 6 outer race faults, and 2 ball faults. Try training anything meaningful with 2 positive samples.
The solution was to collapse ball and inner race faults into a single “rolling element fault” class for the target domain, even though CWRU had them separate. Pragmatically, both require immediate maintenance attention anyway—the distinction matters less than detection.
# Remap labels for target domain
# 0: Normal, 1: Inner race, 2: Outer race, 3: Ball → 0: Normal, 1: Rolling element, 2: Outer race
def remap_labels(y):
mapping = {0: 0, 1: 1, 2: 2, 3: 1} # Ball → Inner race class
return np.array([mapping[label] for label in y])
y_target_remapped = remap_labels(y_target)
print(f"Class distribution: {np.bincount(y_target_remapped)}") # [28, 13, 6]
Still imbalanced, but at least the minority class has 6 samples instead of 2.
FAQ
Q: Can I skip domain adaptation if I fine-tune on a few target samples?
Pure fine-tuning on 47 samples will overfit catastrophically. I tried it—got 95% on the 47 samples and 58% on held-out target data. Domain adaptation techniques like CORAL regularize the adaptation process by constraining how far feature distributions can shift.
Q: How much target data do I really need for cross-machine transfer?
In my experience, 30-50 labeled samples per class is the minimum for CORAL + pseudo-labeling to work. Below 20 samples per class, even domain adaptation struggles. If you have fewer than 10 target samples, consider unsupervised anomaly detection instead of supervised classification.
Q: Does transfer learning work across completely different machine types (e.g., bearings to gearboxes)?
Not in my testing. The fault signatures are fundamentally different. Bearing faults produce characteristic frequencies related to geometry, while gearbox faults produce mesh frequency harmonics and sidebands. Transfer learning works best within the same component type across different operating conditions or sizes.
Debugging domain shift at 2am is a special kind of frustration. If you’re going through the same pain, good noise-canceling headphones help maintain sanity while staring at t-SNE plots.
For cross-machine fault diagnosis, CORAL + pseudo-labeling outperformed MMD and DANN on small target datasets in my tests. The combination of covariance alignment and confident self-training handles the low-data regime better than pure distribution matching. If you have more target data (200+ samples), DANN might catch up, but I haven’t verified that.
What I’m still working on: using physics-informed features (like bearing fault frequency ratios) that transfer naturally across machines, rather than relying entirely on learned features. The fault frequency relationships are determined by geometry—that physics shouldn’t change even when amplitudes and transfer paths do. If I can encode those relationships as constraints, the domain gap might shrink before any adaptation even starts.
Did you find this helpful?
Your support keeps this blog running and ad-free content coming.
☕ Buy me a coffeeMost Popular Posts
- Custom Metaclass in Python: 43% Faster Validation (12,936 views)
- yfinance Alternatives 2026: 7 Free APIs Compared (999 views)
- Python match-case: 7 Patterns That Beat if-elif Chains (977 views)
- YOLOv8 INT8 Quantization: 4x Faster on Jetson Orin (876 views)
- PaddleOCR vs EasyOCR vs Tesseract: Why PaddleOCR Is Slower (690 views)