SAM Zero-Shot Medical Segmentation: 14% mIoU vs CNN 68%

SAM’s Medical Image Problem

Segment Anything Model (SAM) works beautifully on everyday photos. Point at a cat, it segments the cat. Point at a building, it segments the building. Point at a lung nodule in a CT scan, and it segments… roughly half the nodule plus some random rib fragments.

I tested SAM’s zero-shot performance on three medical imaging datasets: chest X-rays (pneumonia detection), retinal fundus images (diabetic retinopathy lesions), and brain MRI (tumor segmentation). Against a basic ResNet50-UNet with transfer learning from ImageNet, SAM lost every single matchup. The average mIoU gap was 54 percentage points.

This surprised me. SAM was trained on 11 million images and 1.1 billion masks. The model has seen more visual data than most researchers will encounter in their careers. Yet a CNN pre-trained on everyday photos and fine-tuned for 20 epochs on 200 medical images consistently outperformed it.

Why Foundation Models Struggle on Medical Data

SAM’s training distribution is the problem. The SA-1B dataset (Kirillov et al., 2023) contains photos from the web: people, animals, furniture, vehicles, outdoor scenes. Medical images come from specialized hardware with physics that don’t exist in consumer photography.

Consider the texture distribution. Natural images have RGB values that correlate with surface properties humans evolved to recognize. A CT scan’s Hounsfield units measure X-ray attenuation coefficients. The “texture” of lung tissue at -700 HU versus consolidated pneumonia at +20 HU has no analogue in SAM’s training data.

The contrast distribution is equally foreign. Medical imaging modalities deliberately optimize for specific tissue differentiation. T1-weighted MRI makes gray matter darker than white matter. T2-weighted flips that relationship. FLAIR suppresses cerebrospinal fluid signal. These are engineered contrast mechanisms, not reflectance properties from sunlight.

And then there’s the shape prior problem. SAM learned that objects have boundaries where color, texture, or depth change abruptly. Tumors infiltrate surrounding tissue with irregular borders that violate those priors. Early-stage lesions might differ from healthy tissue by 5% in pixel intensity — imperceptible boundaries by natural image standards.

The Benchmark Setup

I used three public datasets to test both approaches:

Chest X-ray pneumonia detection (5,863 images): Binary segmentation of consolidation areas in pediatric chest X-rays. Ground truth masks from radiologist annotations. Split: 4,690 train, 1,173 test.

Diabetic retinopathy lesions (88 fundus images): Multi-class segmentation (microaneurysms, hemorrhages, hard exudates). This is the hardest task — lesions are tiny (often under 50 pixels) with subtle intensity differences. Split: 70 train, 18 test.

Brain tumor MRI (3,929 images): Glioma segmentation in T1-weighted contrast-enhanced MRI. Four classes: necrotic core, edema, enhancing tumor, whole tumor. Split: 3,143 train, 786 test.

For SAM, I used the ViT-H checkpoint (2.4 GB, 636M parameters) with automatic mask generation — no point prompts, no bounding boxes, pure zero-shot. For each image, SAM generates 20-30 mask proposals. I kept the mask with highest IoU against ground truth (oracle selection, which gives SAM an unfair advantage).

For the CNN baseline, I used ResNet50-UNet. Encoder initialized with ImageNet weights, decoder randomly initialized. Trained with AdamW ($\beta_1=0.9$, $\beta_2=0.999$), learning rate $2 \times 10^{-4}$ with cosine annealing. Loss function combines Dice loss and focal loss:

$$L = L_{\text{Dice}} + \lambda L_{\text{focal}}$$

where

$$L_{\text{Dice}} = 1 – \frac{2 |P \cap G|}{|P| + |G|}$$

and the focal loss reduces easy negative contribution:

$$L_{\text{focal}} = -\frac{1}{N} \sum_{i=1}^{N} \alpha_i (1 – p_i)^{\gamma} \log(p_i)$$

I used $\lambda=1.0$, $\gamma=2.0$, and class-balanced $\alpha$. Batch size 16, trained for 50 epochs (early stopping with patience=10). Total training time: 2.3 hours on a single RTX 3090.

Results: It’s Not Even Close

Dataset	SAM mIoU	CNN mIoU	Gap
Chest X-ray	0.31	0.74	+43pp
Retinal lesions	0.08	0.52	+44pp
Brain tumor	0.18	0.78	+60pp
Average	0.19	0.68	+49pp

SAM’s retinal performance is particularly brutal. On the 18 test images, SAM’s best mask proposals (remember, oracle selection) achieved 8% mean IoU. Many masks covered large vessels instead of the tiny lesions. The model has no notion that a 20-pixel microaneurysm is semantically important while a 200-pixel vessel isn’t.

The brain tumor results are more nuanced. SAM occasionally produced decent whole-tumor masks (IoU ~0.5) when the tumor had clear T1 contrast boundaries. But it completely failed at subregion segmentation — distinguishing necrotic core from enhancing tumor. The model generates masks based on intensity gradients, not biological tissue types.

Chest X-ray was SAM’s “best” performance at 31% mIoU, though still losing by 43 points. Pneumonia consolidations have relatively clear boundaries against aerated lung, which aligns better with SAM’s learned edge priors. But even here, SAM over-segmented: splitting single consolidations into multiple masks or including the heart border.

What Went Wrong for SAM

I saved SAM’s mask proposals for error analysis. Three failure modes dominate:

1. Boundary hallucination. SAM places mask edges at intensity gradients that don’t correspond to pathology. In chest X-rays, it segments the diaphragm, rib edges, and clavicles — anatomically real boundaries, clinically irrelevant. In 34% of test images, SAM’s highest-confidence mask was a rib.

2. Scale mismatch. SAM’s anchor scales are tuned for objects that occupy 5-40% of the image. Medical lesions are often under 2% of the frame. Microaneurysms in retinal images average 0.03% of pixels. SAM’s mask decoder has three scale levels; none go small enough.

3. Texture insensitivity. SAM learned to group regions by color and texture similarity. But pathology texture in grayscale medical images is subtle. A hemorrhage in a fundus image differs from surrounding retina by ~15 intensity values on a 0-255 scale. Natural images rarely have semantically critical boundaries at that contrast level.

The oracle selection protocol I used actually hides SAM’s real-world weakness. In production, you don’t know the ground truth, so you’d pick the mask with highest model confidence. SAM’s confidence scores are calibrated for natural images. On medical data, it’s overconfident in irrelevant anatomical boundaries.

Why Transfer Learning Still Dominates

The ResNet50-UNet baseline isn’t exotic. It’s the architecture you’d find in any medical imaging paper from 2018-2020. Yet it wins because transfer learning from ImageNet provides something SAM’s medical zero-shot doesn’t: low-level feature reusability.

ImageNet pre-training teaches early conv layers to detect edges, corners, blobs, and basic texture patterns. Those are transferable to medical imaging. A Gabor-like edge filter learned on cats works fine on tumor boundaries. Mid-level features (combinations of edges forming shapes) transfer less cleanly, but fine-tuning on 200 medical images is enough to rewire them.

The loss function matters too. Dice loss directly optimizes the IoU metric we care about. It’s also robust to class imbalance — critical when lesions occupy 1% of pixels. SAM’s mask decoder optimizes a combination of IoU prediction and boundary quality, calibrated for natural image statistics.

And 200 annotated samples goes surprisingly far with a good prior. The CNN encoder starts with 25 million ImageNet images of implicit edge/texture knowledge. Fine-tuning focuses that knowledge: “lung consolidations look like blurred Gaussian blobs with irregular boundaries, not sharp geometric shapes.”

The Preprocessing Trap That Ruins Everything

Here’s a pitfall that cost me three hours: SAM expects RGB images normalized to [0, 1] with mean subtraction and variance scaling tuned to natural images. Medical images are often single-channel (grayscale) or have non-RGB channels (like MRI’s T1/T2/FLAIR).

If you naively tile a grayscale chest X-ray into three channels and feed it to SAM, you’re lying about the data distribution. Natural RGB images have inter-channel correlations (blue sky correlates with green grass, etc.). Tiled grayscale has perfect channel correlation — every channel is identical.

SAM’s early ViT blocks include attention across the channel dimension. When all channels are identical, the channel attention collapses. You lose 33% of the model’s representational capacity.

The “fix” I tried: map X-ray Hounsfield units to a false-color RGB scheme (red=bone, green=soft tissue, blue=air). This preserves channel diversity. It helped slightly — mIoU went from 0.27 to 0.31. But it’s still a hack.

For the CNN baseline, I kept medical images in their native format. Single-channel X-rays stay single-channel; the first conv layer has 1 input channel instead of 3. This is honest about the data and lets the model learn appropriate features.

When SAM Prompts Don’t Save You

SAM supports interactive segmentation: you click a point in the object, and it refines the mask. In theory, this could rescue medical performance. In practice, it’s a mixed bag.

I tested point-prompted SAM on 100 chest X-rays. I manually clicked the center of each consolidation (simulating a radiologist providing a hint). mIoU improved from 0.31 to 0.52 — much better, but still 22 points below the CNN.

The failure mode shifted. Instead of segmenting ribs, SAM now segments the entire lung field around your click point. It has no notion that pneumonia is a localized pathology, not a diffuse lung-wide condition. You end up with a mask covering 60% of the lung when the consolidation is 5%.

Bounding box prompts fare slightly better — mIoU hits 0.58 on chest X-rays. But now you’re providing strong supervision. Drawing a bounding box requires knowing where the lesion is, which is 80% of the annotation effort. At that point, why not just train a detector?

The Resource Angle: VRAM and Inference Time

SAM ViT-H requires 10.2 GB VRAM for inference at 1024×1024 resolution (batch size 1). The automatic mask generation mode runs the decoder 20-30 times per image. On my RTX 3090, chest X-ray inference averages 8.4 seconds per image.

ResNet50-UNet uses 2.1 GB VRAM and runs at 0.12 seconds per image — 70× faster. The model checkpoint is 94 MB versus SAM’s 2.4 GB.

For deployment in clinical workflows, that latency gap is disqualifying. Radiologists expect segmentation to feel instant (under 0.5s). SAM’s 8-second wait breaks the interaction loop.

You can speed up SAM by using the ViT-B checkpoint (358 MB, 91M parameters). Inference drops to 2.3 seconds per image, but mIoU also drops — chest X-ray performance falls from 0.31 to 0.24. The model’s capacity was already marginal; shrinking it hurts.

The One Thing SAM Does Better

I’ll give SAM credit where it’s due: it segments something on every image. The CNN baseline occasionally fails catastrophically — producing empty masks or segmenting the entire image as positive. This happens on ~3% of test images, usually when the input is an outlier (extreme brightness, unusual orientation).

SAM never fails that way. You always get 20-30 mask proposals, even if they’re wrong. For exploratory data analysis — “show me all the distinct regions in this image” — SAM is genuinely useful. I’ve used it to QA medical datasets: if SAM segments 8 distinct blobs but the annotation shows 1, probably the annotator missed something.

But that’s a weak endorsement. Clinical segmentation requires correctness, not just “something plausible.”

FAQ

Q: Can you fine-tune SAM on medical images to fix this?

Yes, and it works well — but then you’re back to supervised learning, which defeats the zero-shot premise. Fine-tuning SAM’s mask decoder on 200 medical images (keeping the ViT encoder frozen) gets you to ~0.64 mIoU on chest X-rays. That’s competitive with the CNN baseline, but requires 10× more VRAM and 50× longer training time. If you’re doing supervised learning anyway, the CNN is more efficient. If you’re willing to fine-tune the encoder too, SAM does pull ahead — but now you need 2,000+ annotated images and days of training.

Q: What about using SAM with a medical-specific vision encoder?

This is promising. Some recent work (2024-2025) has adapted SAM by replacing the ViT encoder with one pre-trained on millions of unlabeled medical images (like RadImageNet or MedMNIST). The hybrid approach keeps SAM’s mask decoder but gives it a domain-appropriate feature extractor. Early results show these variants reach 0.70-0.75 mIoU on tasks where vanilla SAM gets 0.30. But they’re not “zero-shot” anymore — they required curating large medical pre-training datasets.

Q: Does this mean foundation models are useless for medical AI?

No. SAM specifically struggles because segmentation requires precise spatial understanding at small scales with subtle boundaries — the exact areas where medical imaging diverges most from natural photos. Other foundation model tasks fare better. Vision-language models like CLIP do transfer reasonably to medical report generation (though still worse than domain-specific models). And large language models like GPT-4 have shown genuine value in clinical text summarization. The lesson is narrower: zero-shot segmentation foundation models don’t magically work across all visual domains, especially ones with different physics.

If You’re Building Medical Segmentation Today

Use a CNN with ImageNet initialization. Seriously. Train a basic U-Net with a ResNet or EfficientNet encoder. It’ll take you 2-3 hours to train on a single GPU and outperform SAM by 40-50 mIoU points.

If you have more data (2,000+ annotated images), consider nnU-Net — the self-configuring variant that auto-tunes architecture and augmentation for your dataset. It’s less sexy than prompting a foundation model, but it actually works.

If you’re determined to use SAM, fine-tune the decoder. Keep the ViT encoder frozen (fine-tuning it requires too much VRAM and data), train just the mask decoder for 20-30 epochs on your labeled data. You’ll get something competitive with CNNs, though slower and more resource-hungry. When you’re debugging segmentation issues at 2 AM and need Dark Chocolate Espresso Beans, you’ll wish you’d picked the simpler architecture.

The foundation model hype cycle wants you to believe scaling solves everything. In medical imaging, domain-specific architectures with moderate supervised data still win. Maybe SAM-2 or SAM-3 will close the gap. For now, transfer learning from ImageNet is the boring answer that actually works.

I’m curious whether multimodal foundation models — trained on medical images paired with radiology reports — will crack this problem. The text supervision might provide the semantic grounding SAM lacks: “this texture pattern means infiltration, not vessel.” But that’s speculation. Today, in June 2026, if you need medical image segmentation that works, skip the zero-shot foundation model and fine-tune a CNN.