Tag: Dynamic Routing
-
Mixture-of-Depths: Dynamic Token Skip Cuts 40% FLOPs
MoD routing lets tokens skip layers dynamically, cutting 40% FLOPs in GPT-scale models without quality loss. Here's how it works and when it breaks.
MoD routing lets tokens skip layers dynamically, cutting 40% FLOPs in GPT-scale models without quality loss. Here's how it works and when it breaks.