Tag: Sparse Routing
-
Switch Transformers ๋ฆฌ๋ทฐ: 1.6์กฐ ํ๋ผ๋ฏธํฐ MoE ๋ชจ๋ธ
Switch Transformers paper decoded: how Top-1 routing scales MoE to 1.6T parameters, load balancing tricks, and distillation speed hacks.
Switch Transformers paper decoded: how Top-1 routing scales MoE to 1.6T parameters, load balancing tricks, and distillation speed hacks.