Tag: inference optimization
-
Test-Time Augmentation in Production: 3x Slower, 1.2% Better
TTA promises better accuracy but costs 5x GPU budget. Real benchmarks from defect detection, medical imaging, and ImageNet show when it's worth it.
-
Ollama vs llama.cpp vs vLLM: Throughput on M1/RTX 4090
Compare Ollama, llama.cpp, and vLLM throughput on M1 and RTX 4090. Discover which framework delivers the best performance for local LLM inference.
-
SAM 2 Inference Pipeline Bottlenecks: 3x Slower Than SAM
SAM 2 inference runs 3x slower than SAM despite better accuracy. Discover the bottleneck causing this performance drop and how to optimize it.
-
Ollama vs vLLM vs llama.cpp: Which Wins for Your Use Case
vLLM hits 47x higher throughput than Ollama at 32 concurrent requests. Real benchmarks reveal when each framework wins โ and the memory tradeoffs nobody mentions.
-
LLM Memory Calculator: Online Estimators Miss 40% Usage
Calculate LLM memory needs accurately. Why online tools fail at KV cache estimation and how to fix it with real GPU profiling methods.
-
Speculative Decoding vs MoE: 3.2x Cost Gap on Llama 3
Compare Speculative Decoding vs MoE on Llama 3. Discover why one costs 3.2x more and which inference optimization truly delivers better value.
-
GPU vs CPU Inference: 5 Scenarios, Real Costs & Latency
GPU vs CPU inference across 5 traffic scenarios: real costs, latency benchmarks, and when each makes sense. BERT/ResNet/Whisper tested on AWS.
-
TorchServe vs ONNX Runtime: First Inference in 5 Minutes
Compare TorchServe vs ONNX Runtime deployment speed, memory usage, and ease of setup. See which inference server wins in real-world performance tests.