Category: Paper Review
-
Switch Transformers 리뷰: 1.6조 파라미터 MoE 모델
Switch Transformers paper decoded: how Top-1 routing scales MoE to 1.6T parameters, load balancing tricks, and distillation speed hacks.
-
Mamba 리뷰: 선형 시간복잡도 State Space Model
Mamba hits O(L) complexity vs Transformer's O(L²). Selective State Space Model achieves 5× faster inference on DNA and audio with benchmarks.
-
FlashAttention 리뷰: Transformer 어텐션 2~4배 가속
FlashAttention cuts Transformer memory from O(N²) to O(N) while running 2-4× faster. Here's the IO-aware tiling trick explained.
-
RetNet 논문 리뷰: Transformer를 대체할 병렬 학습+순차 추론 아키텍처 완전 분석
RetNet achieves 8.4× faster inference than Transformer with 70% less memory. The parallel training + sequential inference proof inside.
-
[논문리뷰] Mamba: Selective State Space Model로 Transformer의 한계를 돌파하다
Mamba beats Transformer with linear-time complexity using Selective State Space Models. See the selection mechanism that makes it work.
-
DeepSeek-V3 논문 리뷰: MoE 기반 오픈소스 LLM이 GPT-4o를 넘어선 비결
DeepSeek-V3 matches GPT-4o with 671B parameters but activates only 37B. The MoE, MLA, and load balancing techniques that made it possible.