Tag: FlashAttention
-
FlashAttention-2 vs xFormers: H100 Cost at 100M Tokens
Compare FlashAttention-2 vs xFormers on H100 GPUs for 100M token training. Discover which framework cuts costs and boosts speed for LLM workloads.
-
FlashAttention-2 Warmup: Fix 3x Slower First Batch
First FlashAttention-2 batch is 3x slower? Fix kernel compilation overhead with warmup, persistent cache, and bucketingโreal latency numbers included.
-
FlashAttention: Transformer ๋ฉ๋ชจ๋ฆฌ ์ต์ ํ ์ค์ ๊ตฌํ
FlashAttention-3 cuts Transformer memory from O(nยฒ) to O(n) on H100 GPUs. See PyTorch benchmarks proving 3-5x speedup on 32k context windows.
-
FlashAttention ๋ฆฌ๋ทฐ: Transformer ์ดํ ์ 2~4๋ฐฐ ๊ฐ์
FlashAttention cuts Transformer memory from O(Nยฒ) to O(N) while running 2-4ร faster. Here's the IO-aware tiling trick explained.