Tag: inference optimization
-
ONNX Runtime Inlining Flags: 8x Latency Cut in 4 Steps
Cut ONNX Runtime latency by 8x with 4 inlining flags. Learn session config tuning for C++/Python inference optimizationโsimple yet powerful.
-
FlashAttention-2 Warmup: Fix 3x Slower First Batch
First FlashAttention-2 batch is 3x slower? Fix kernel compilation overhead with warmup, persistent cache, and bucketingโreal latency numbers included.
-
FastAPI Model Serving: 5 Steps to 50ms Inference
Cut FastAPI model inference from 800ms to 47ms with 5 changes: lifespan handlers, warmup, ThreadPoolExecutor, Pydantic validation, and response models.