Tag: mixed precision
-
Mixed Precision Training NaN Loss: 4 Root Causes & Fixes
Fix 4 FP16 training bugs that cause NaN loss: gradient scale overflow, unstable loss functions, norm variance explosion, and accumulation errors.
-
Transformer NaN Loss: 7 Fixes That Actually Work
Transformer NaN loss? Don't lower learning rate first. Fix attention overflow, fp16 underflow, and LayerNorm edge cases with these 7 patches.