Tag: Router Collapse
-
MoE Router Collapse: Why 90% of Tokens Hit 2 Experts
87% of tokens routing to 2 experts? That's router collapse killing your MoE model. Here's the auxiliary loss fix and diagnostic code to catch it early.
87% of tokens routing to 2 experts? That's router collapse killing your MoE model. Here's the auxiliary loss fix and diagnostic code to catch it early.