Tag: Transformer Optimization
-
Mixture-of-Depths: Dynamic Token Skip Cuts 40% FLOPs
MoD routing lets tokens skip layers dynamically, cutting 40% FLOPs in GPT-scale models without quality loss. Here's how it works and when it breaks.
-
GQA Review: Grouped Query Attention for Faster LLM Inference
GQA reduces LLM inference memory 8x with only 0.1 ROUGE loss. How grouped KV heads and 5% uptraining work in Llama 2 and Mistral.