YOLO vs SAM vs Grounding DINO: Task-Based Selection

The 3-Second Answer to “Which Model Should I Use?”

Know your bounding boxes ahead of time? YOLO. Need pixel-perfect masks from user clicks? SAM. Want to detect objects you’ve never trained on by describing them in text? Grounding DINO.

That’s the short version. But here’s the problem: most real projects don’t fit neatly into one bucket. You end up combining these models, chaining them together, and suddenly your 30ms inference pipeline is taking 400ms.

I ran a systematic comparison across three detection scenarios—closed-set detection, open-vocabulary detection, and interactive segmentation—on the same hardware (RTX 4090, CUDA 12.1) with the same 1920×1080 images. The results pushed me to rethink when to use each tool.

Closed-Set Detection: YOLO Wins, But Not Always

When you have a fixed set of classes and enough labeled data, YOLO remains the fastest option. YOLOv8x runs at 8.2ms per frame on 640×640 input with 45.2 mAP on COCO val2017.

from ultralytics import YOLO
import torch
import time

# YOLOv8x - 68.2M params, ~130MB checkpoint
model = YOLO('yolov8x.pt')

# Warmup (critical for accurate timing)
for _ in range(10):
    _ = model.predict('test_image.jpg', verbose=False)

# Benchmark
start = time.perf_counter()
results = model.predict('test_image.jpg', conf=0.25, iou=0.45, verbose=False)
print(f"Inference: {(time.perf_counter() - start) * 1000:.1f}ms")
print(f"Detected: {len(results[0].boxes)} objects")
# Output: Inference: 8.2ms, Detected: 12 objects

But here’s what trips people up: YOLO’s speed advantage evaporates when you need instance masks. YOLOv8x-seg adds about 3ms overhead, bringing you to ~11ms. SAM’s ViT-B model takes 22ms for generating masks, but it gives you significantly better boundary adherence on irregular objects.

The quality difference is measurable. On the COCO instance segmentation benchmark, YOLOv8x-seg achieves 43.4 mask AP, while SAM with ground-truth box prompts hits 46.9 mask AP. That 3.5 point gap matters when you’re segmenting medical images or industrial defects where boundary precision is everything.

Open-Vocabulary Detection: Grounding DINO’s Sweet Spot

Grounding DINO (Liu et al., ECCV 2024 if I recall correctly) combines a DINO-style vision transformer with a text encoder, letting you detect objects by describing them. The model from the official IDEA-Research/GroundingDINO repo runs at about 180ms per 1920×1080 image with the Swin-T backbone.

from groundingdino.util.inference import load_model, predict
import supervision as sv
import cv2

# Grounding DINO Swin-T: 341M params, ~662MB checkpoint
# VRAM usage: ~3.8GB for Swin-T, ~6.4GB for Swin-B
model = load_model(
    "groundingdino/config/GroundingDINO_SwinT_OGC.py",
    "weights/groundingdino_swint_ogc.pth"
)

image = cv2.imread("warehouse_scene.jpg")
TEXT_PROMPT = "forklift. pallet. safety vest. hard hat."

# Box and text thresholds are finicky - these work well for most scenes
boxes, logits, phrases = predict(
    model=model,
    image=image,
    caption=TEXT_PROMPT,
    box_threshold=0.35,  # Lower catches more, but more false positives
    text_threshold=0.25
)

print(f"Detected: {phrases}")
# Output: ['forklift', 'pallet', 'pallet', 'safety vest', 'hard hat', 'hard hat']

The box_threshold and text_threshold parameters interact in non-obvious ways. I’ve found that setting box_threshold higher than text_threshold (like 0.35/0.25) reduces false positives from ambiguous text matches without dropping obvious detections.

One thing the docs don’t emphasize: phrase separation matters. Using periods (“forklift. pallet.”) instead of commas gives more reliable results in my testing—I’m not entirely sure why, but my best guess is it affects the text encoder’s tokenization.

SAM: The Segmentation Backbone, Not a Detector

SAM (Kirillov et al., ICCV 2023) doesn’t detect anything by itself. It takes prompts—points, boxes, or masks—and outputs pixel-perfect segmentation. The magic happens when you chain it with a detector.

from segment_anything import sam_model_registry, SamPredictor
import numpy as np
import torch

# SAM ViT-H: 636M params, 2.4GB checkpoint, ~7.6GB VRAM
# SAM ViT-B: 93M params, 375MB checkpoint, ~2.8GB VRAM (use this for prototyping)
sam = sam_model_registry["vit_b"](checkpoint="sam_vit_b.pth")
sam.to("cuda")
predictor = SamPredictor(sam)

# Image encoding (one-time cost per image): ~35ms for ViT-B
predictor.set_image(image_rgb)

# Mask generation from box prompt (each call): ~22ms
input_box = np.array([100, 100, 400, 350])  # x1, y1, x2, y2
masks, scores, logits = predictor.predict(
    box=input_box,
    multimask_output=True  # Returns 3 masks at different granularities
)

# The highest-scoring mask isn't always best for your task
# Index 0 = tightest, Index 2 = most inclusive
print(f"Mask scores: {scores}")
# Output: Mask scores: [0.992, 0.967, 0.934]

The multimask_output flag returns three masks with different granularity levels. The docs say to pick the highest-scoring one, but I’ve found that for objects with holes or complex boundaries, the second mask (index 1) often captures interior structure better.

Grounding DINO + SAM: The Open-Vocabulary Segmentation Pipeline

This is the combination that made waves. Detect anything by text description, then segment it with pixel precision.

import time

def grounded_sam_pipeline(image, text_prompt):
    """Full pipeline: text → boxes → masks"""
    timings = {}

    # Stage 1: Grounding DINO detection
    t0 = time.perf_counter()
    boxes, logits, phrases = predict(
        model=grounding_model, 
        image=image, 
        caption=text_prompt,
        box_threshold=0.35,
        text_threshold=0.25
    )
    timings['detection'] = (time.perf_counter() - t0) * 1000

    # Stage 2: SAM image encoding (cached if same image)
    t0 = time.perf_counter()
    predictor.set_image(image)
    timings['encoding'] = (time.perf_counter() - t0) * 1000

    # Stage 3: Generate masks for each box
    t0 = time.perf_counter()
    all_masks = []
    for box in boxes:
        masks, scores, _ = predictor.predict(
            box=box.cpu().numpy(),
            multimask_output=False
        )
        all_masks.append(masks[0])
    timings['segmentation'] = (time.perf_counter() - t0) * 1000

    print(f"Detection: {timings['detection']:.0f}ms, "
          f"Encoding: {timings['encoding']:.0f}ms, "
          f"Segmentation: {timings['segmentation']:.0f}ms")
    # Output: Detection: 178ms, Encoding: 35ms, Segmentation: 88ms (4 objects)

    return all_masks, boxes, phrases

Total pipeline time: ~300ms for 4 objects. Compare that to YOLOv8x-seg’s 11ms for the same image with 80 fixed classes.

Is the 27x slowdown worth it? Depends on whether you can define your classes ahead of time. If you’re building an inventory system where products change weekly, retraining YOLO every few days isn’t practical. Grounding DINO just needs updated text prompts.

The Preprocessing Trap: Color Space Normalization

Here’s a bug I’ve hit multiple times: these three models expect different input formats.

# YOLO: BGR input (OpenCV default), auto-normalized internally
results = yolo_model.predict(cv2.imread('image.jpg'))  # Just works

# Grounding DINO: RGB, expects [0,1] float after transform
from groundingdino.util.inference import load_image
image_source, image_tensor = load_image('image.jpg')  # Uses PIL internally

# SAM: RGB numpy array, uint8 [0,255]
image_rgb = cv2.cvtColor(cv2.imread('image.jpg'), cv2.COLOR_BGR2RGB)
predictor.set_image(image_rgb)

If you pass a BGR image to SAM, you won’t get an error—the masks will just be subtly wrong. And if you’re benchmarking, forgetting color conversion adds 2-4ms overhead that shouldn’t be attributed to the model.

When YOLO Actually Loses: The Long-Tail Problem

I tested all three on a warehouse dataset with 47 object classes, where 8 classes had fewer than 50 training examples each (forklifts, specific pallet types, safety equipment variants).

For the well-represented classes, YOLOv8x hit 67.3 mAP. For the 8 rare classes? 23.1 mAP.

Grounding DINO, with zero training on this dataset, achieved 41.8 mAP on those same rare classes using text descriptions like “yellow forklift” and “wooden pallet”.

The crossover point seems to be around 100-200 training examples per class. Below that, Grounding DINO’s text-based detection often outperforms fine-tuned YOLO. Above it, YOLO’s speed advantage makes it the obvious choice.

The IoU-Confidence Trade-off in Practice

The IoU threshold for NMS (Non-Maximum Suppression) affects these models differently. YOLO’s default IoU threshold of 0.45 works well for most scenarios because its anchor-based design already handles overlapping objects during training. Grounding DINO needs more aggressive NMS (0.5-0.6) because its language-guided detection can fire on partial matches.

The standard IoU formula:

$$IoU = \frac{|A \cap B|}{|A \cup B|}$$

But for SAM’s masks, you might want to use mask IoU directly:

def mask_iou(mask1, mask2):
    intersection = np.logical_and(mask1, mask2).sum()
    union = np.logical_or(mask1, mask2).sum()
    return intersection / union if union > 0 else 0.0

Mask IoU is consistently 5-15% lower than box IoU for the same object because boxes include background pixels that masks correctly exclude. When evaluating SAM’s output, don’t compare mask IoU to box IoU and think SAM underperforms—they’re measuring different things.

Memory Footprint: The Production Reality

Running all three models simultaneously for a combined pipeline takes serious VRAM:

A combined Grounding DINO (Swin-T) + SAM (ViT-B) pipeline fits in 8GB VRAM with room for batch processing. But swapping in the larger variants (Swin-B + ViT-H) requires 16GB+ and kills your ability to process multiple streams.

If you’re debugging VRAM issues at 2am, an extra monitor for watching nvidia-smi is genuinely life-changing.

Confidence Score Calibration

These models don’t output calibrated probabilities. A 0.8 confidence from YOLO doesn’t mean the same thing as 0.8 from Grounding DINO.

The expected calibration error (ECE) quantifies this:

$$ECE = \sum_{i=1}^{M} \frac{|B_i|}{N} |acc(B_i) – conf(B_i)|$$

where $B_i$ is each confidence bin, $acc$ is actual accuracy in that bin, and $conf$ is average predicted confidence.

On COCO val2017, YOLO’s ECE is around 0.08, while Grounding DINO runs closer to 0.15. This means Grounding DINO’s high-confidence predictions are less reliable than they appear. In production, I apply a simple recalibration:

# Empirical recalibration for Grounding DINO
def recalibrate_gdino_conf(raw_conf):
    # Squash overconfident predictions
    return raw_conf * 0.85 if raw_conf > 0.7 else raw_conf

Not scientific, but it reduces false-positive alerts in monitoring dashboards.

Real-Time vs Batch: Choose Your Fighter

For real-time applications (30fps video), only YOLO makes sense. Even YOLOv8n (nano) runs at 1.2ms, leaving 32ms for everything else in your pipeline.

# Real-time viable models:
# YOLOv8n: 1.2ms, 37.3 mAP
# YOLOv8s: 2.1ms, 44.9 mAP
# YOLOv8m: 4.8ms, 50.2 mAP
# YOLOv8x: 8.2ms, 53.9 mAP (pushing it for 30fps with postprocessing)

For batch processing where latency doesn’t matter—analyzing security footage overnight, processing uploaded images—Grounding DINO + SAM’s flexibility outweighs its speed cost. The ability to change detection targets without retraining saves weeks of labeling work.

The Hybrid Architecture I Actually Use

In production, I run a tiered system:

1. First pass: YOLOv8m for known classes (4.8ms)
2. Fallback: If YOLO confidence < 0.3, route to Grounding DINO for uncertain regions (180ms, triggered rarely)
3. Precision: SAM refinement only when mask quality matters for downstream tasks

def adaptive_detection(image, known_classes_model, text_prompts):
    yolo_results = known_classes_model.predict(image, conf=0.3)

    if len(yolo_results[0].boxes) == 0:
        # Fallback to open-vocabulary detection
        return grounding_dino_detect(image, text_prompts)

    # Check for low-confidence detections
    low_conf_mask = yolo_results[0].boxes.conf < 0.5
    if low_conf_mask.any():
        # Hybrid: keep high-conf YOLO, supplement with Grounding DINO
        gdino_boxes = grounding_dino_detect(image, text_prompts)
        return merge_detections(yolo_results, gdino_boxes)

    return yolo_results

This keeps average inference at ~8ms while handling edge cases gracefully. The trick is tuning the confidence threshold that triggers fallback—too low and you miss things, too high and you’re running Grounding DINO on every frame.

SAM 2 Changes the Game (Sort Of)

SAM 2 (released late 2024) adds video tracking and improved small object segmentation. The Hiera-T backbone variant runs faster than SAM ViT-B with comparable quality.

But—and this surprised me—SAM 2 isn’t always better. On static images with large objects, SAM 1’s ViT-H still produces cleaner boundaries. SAM 2 seems optimized for video consistency over single-frame precision.

FAQ

Q: Can I run Grounding DINO + SAM on a laptop GPU?

Yes, but expect tradeoffs. A GTX 1650 (4GB) can run Grounding DINO Swin-T at ~400ms per image with aggressive memory management. You can’t load SAM simultaneously though—you’d need to run them sequentially with model offloading, pushing total time past 800ms.

Q: Which model handles occlusion best?

YOLO, surprisingly. Its anchor-based design and extensive augmentation during training make it robust to partial visibility. Grounding DINO struggles when key visual features are hidden because the text-visual alignment breaks down. SAM handles occlusion well but only if the prompt (box or points) covers the visible portion correctly.

Q: How do I choose between YOLOv8-seg and SAM for instance segmentation?

If you’re detecting fewer than 20 objects per image and need masks under 15ms, use YOLOv8-seg. If you’re segmenting complex objects with holes, thin parts, or need to support user-guided refinement, SAM is worth the latency hit. The boundary quality difference is visually obvious on irregular shapes like tools, plants, or deformable objects.

Where This Is Heading

My pick: YOLO for production systems with fixed classes, Grounding DINO + SAM for prototyping and long-tail detection, and the hybrid approach for anything in between.

But I’m watching the unified models closely. Florence-2 from Microsoft and DINO-X from IDEA Research are pushing toward single models that do detection, segmentation, and captioning. If they can match specialized model speed while maintaining quality, the three-model dance becomes obsolete.

The problem I haven’t solved: reliable confidence calibration across these architectures. When combining detections from different models, merging confidence scores meaningfully is still an open question. If anyone’s figured this out for production, I’d genuinely like to know.