Tag: CUDA
-
vLLM OutOfMemoryError with Llama 3.1 70B: 3 Fixes
Fix vLLM OutOfMemoryError when deploying Llama 3.1 70B with tensor parallelism, quantization, and KV cache tuning on multi-GPU setups.
-
FlashAttention-2 Warmup: Fix 3x Slower First Batch
First FlashAttention-2 batch is 3x slower? Fix kernel compilation overhead with warmup, persistent cache, and bucketingโreal latency numbers included.