Tag: Multi-Head Attention
-
GQA Review: Grouped Query Attention for Faster LLM Inference
GQA reduces LLM inference memory 8x with only 0.1 ROUGE loss. How grouped KV heads and 5% uptraining work in Llama 2 and Mistral.
GQA reduces LLM inference memory 8x with only 0.1 ROUGE loss. How grouped KV heads and 5% uptraining work in Llama 2 and Mistral.