DETR vs Faster R-CNN: End-to-End Detection Hits 42 AP

The Real Surprise: No NMS, No Anchors, Same Accuracy

Faster R-CNN has dominated object detection since 2015. Anchors, region proposals, non-maximum suppression (NMS)—these handcrafted components became so standard that nobody questioned them. Then Facebook AI dropped DETR in 2020 and achieved 42 AP on COCO with none of that machinery.

You can read the full paper here.

The key insight isn’t just “Transformers work for detection.” It’s that the entire detection pipeline—from feature extraction to final bounding boxes—can be reformulated as a direct set prediction problem. One forward pass, 100 learned queries, bipartite matching loss. Done.

Why Faster R-CNN’s Pipeline Got So Complicated

Before diving into DETR’s elegance, let’s appreciate what it replaced. Faster R-CNN (Ren et al., NeurIPS 2015) needs:

1. Anchor generation: ~15K anchors per image across multiple scales and aspect ratios
2. Region Proposal Network (RPN): First-stage filtering to ~2000 proposals
3. RoI pooling/align: Extract fixed-size features from variable regions
4. Two-stage classification: Separate objectness and class prediction
5. NMS post-processing: Remove duplicate detections (typically IoU threshold 0.5)

Each component requires hyperparameters. Anchor sizes, aspect ratios, NMS thresholds, RPN/detection head ratios—you’re tuning a dozen knobs before training even starts. And these choices don’t transfer well between datasets.

The deeper problem? NMS is fundamentally incompatible with end-to-end learning. You can’t backpropagate through a hard thresholding operation. This disconnect between training and inference has haunted two-stage detectors for years.

DETR’s Core Architecture: Bipartite Matching Kills NMS

DETR’s architecture is deceptively simple. A CNN backbone (typically ResNet-50) extracts features, which get flattened and passed through a Transformer encoder-decoder. The decoder takes 100 learned “object queries” and outputs 100 predictions—always exactly 100, regardless of how many objects exist in the image.

The magic happens in the loss function. Instead of matching predictions to ground truth by IoU proximity (which creates the duplicate detection problem NMS solves), DETR uses Hungarian matching to find the optimal one-to-one assignment between predictions and targets.

The Hungarian algorithm finds the bipartite matching that minimizes total cost:

$$\hat{\sigma} = \arg\min_{\sigma \in \mathfrak{S}_N} \sum_{i}^{N} \mathcal{L}_{\text{match}}(y_i, \hat{y}_{\sigma(i)})$$

where $\mathfrak{S}_N$ is the set of all permutations of $N$ elements, and the matching cost combines classification and box regression:

$$\mathcal{L}_{\text{match}}(y_i, \hat{y}_{\sigma(i)}) = -\mathbb{1}_{\{c_i \neq \varnothing\}} \hat{p}_{\sigma(i)}(c_i) + \mathbb{1}_{\{c_i \neq \varnothing\}} \mathcal{L}_{\text{box}}(b_i, \hat{b}_{\sigma(i)})$$

Once matched, the loss is a weighted combination of negative log-likelihood for classification and box losses:

$$\mathcal{L}_{\text{Hungarian}}(y, \hat{y}) = \sum_{i=1}^{N} \left[ -\log \hat{p}_{\hat{\sigma}(i)}(c_i) + \mathbb{1}_{\{c_i \neq \varnothing\}} \mathcal{L}_{\text{box}}(b_i, \hat{b}_{\hat{\sigma}(i)}) \right]$$

The box loss uses a combination of L1 and generalized IoU:

$$\mathcal{L}_{\text{box}} = \lambda_{\text{iou}} \mathcal{L}_{\text{giou}}(b_i, \hat{b}_{\sigma(i)}) + \lambda_{\text{L1}} \| b_i – \hat{b}_{\sigma(i)} \|_1$$

This formulation guarantees each ground truth gets matched to exactly one prediction. No duplicates means no NMS.

The Encoder-Decoder Attention Dance

The Transformer encoder processes the flattened CNN features with self-attention. With a typical feature map of 25×34 (for 800×1066 input), that’s 850 spatial positions attending to each other. Positional encodings are added—fixed sinusoidal, not learned—so the model knows where each patch came from.

The decoder is where it gets interesting. Each of the 100 object queries attends to the encoded image features through cross-attention. But here’s what tripped me up initially: the queries also attend to each other through self-attention.

Why does query-to-query attention matter? It lets the model reason about relationships between detected objects. If query #37 is confidently detecting a person’s face, query #38 should probably not also detect that same face. The self-attention provides an implicit communication channel for coordinating predictions.

import torch
import torch.nn as nn

class DETRDecoder(nn.Module):
    def __init__(self, d_model=256, nhead=8, num_layers=6):
        super().__init__()
        self.layers = nn.ModuleList([
            DETRDecoderLayer(d_model, nhead) 
            for _ in range(num_layers)
        ])
        self.num_queries = 100
        self.query_embed = nn.Embedding(self.num_queries, d_model)

    def forward(self, memory, pos_embed):
        # memory: encoder output [batch, seq_len, d_model]
        # query_embed: learned object queries [100, d_model]
        batch_size = memory.shape[0]

        # Initialize target with zeros, queries provide positional info
        tgt = torch.zeros(batch_size, self.num_queries, memory.shape[-1], 
                         device=memory.device)
        query_pos = self.query_embed.weight.unsqueeze(0).repeat(batch_size, 1, 1)

        for layer in self.layers:
            # Self-attention among queries, then cross-attention to memory
            tgt = layer(tgt, memory, query_pos, pos_embed)

        return tgt  # [batch, 100, d_model]

I’m not entirely sure why the authors chose 100 queries specifically. The paper mentions it’s “significantly larger than the typical number of objects in an image,” but 50 or 200 would presumably work too. My best guess is that 100 provides enough capacity for dense scenes while keeping memory reasonable.

The Painful Reality: 500 Epochs to Converge

Here’s the number that doesn’t make the headline: DETR needs 500 epochs to converge on COCO, compared to 36 epochs for Faster R-CNN under the same 1x schedule.

That’s roughly 14x more training time.

The authors attribute this to the global attention mechanism needing longer to learn spatial relationships that CNNs capture immediately through local receptive fields. But I think there’s another factor—the Hungarian matching creates a highly non-stationary training signal. Early in training, the random query-to-target assignments are essentially noise. The model needs many epochs just to stabilize which queries attend to which image regions.

Training on 8 V100s, batch size 64. The DC5 variant uses dilated convolutions in stage 5 for higher resolution features, but the memory cost is brutal—you’ll need 32GB GPUs or gradient checkpointing.

Small Objects: Where DETR Falls Apart

The COCO evaluation splits AP by object size: small (<32² pixels), medium (32²-96²), and large (>96²). Here’s where DETR’s numbers get uncomfortable:

DETR crushes large objects (+9.1 AP_L) but loses badly on small ones (-3.7 AP_S).

Why? The global attention mechanism downsamples features 32x before the Transformer sees them. A 32×32 pixel object becomes a single feature vector. Faster R-CNN’s FPN pyramid preserves multi-scale information explicitly, while DETR relies entirely on the attention mechanism to recover spatial precision.

Deformable DETR (Zhu et al., ICLR 2021) later addressed this with multi-scale deformable attention, but that’s a story for another post.

The Ablation That Surprised Me Most

The paper includes extensive ablations. The one that caught my attention: removing the encoder entirely drops AP by only 3.9 points (42.0 → 38.1).

Think about that. Six Transformer encoder layers—36M attention operations per image—contribute less than 4 AP. The decoder alone, with its object queries and cross-attention, does most of the heavy lifting.

This finding led to later work on encoder-free detectors, but it also raises a question the paper doesn’t fully answer: what exactly is the encoder learning? The authors show encoder attention patterns that look like instance segmentation masks, but 3.9 AP seems like a steep price for what’s essentially implicit segmentation.

Positional Encodings: Fixed Beats Learned

Counterintuitively, fixed sinusoidal position encodings outperformed learned embeddings by 0.8 AP. This surprised me—learned embeddings should have strictly more capacity.

The sinusoidal encoding for position $(x, y)$ uses:

$$PE_{(x,y,2i)} = \sin\left(\frac{x}{10000^{2i/d}}\right), \quad PE_{(x,y,2i+1)} = \cos\left(\frac{x}{10000^{2i/d}}\right)$$

with separate encodings for x and y concatenated. The authors hypothesize that fixed encodings generalize better because they don’t overfit to training image sizes. But I suspect the real reason is related to the convergence problem—learned encodings need more epochs to stabilize, and 500 might not be enough.

What Would Make Me Actually Use DETR

Would I deploy DETR in production today? Probably not, unless I specifically needed:

1. End-to-end training with custom losses: DETR’s direct set prediction makes it trivial to add auxiliary losses without breaking NMS assumptions
2. Large object detection: If your use case is detecting cars, furniture, or people (not their faces), DETR’s AP_L advantage is significant
3. Panoptic segmentation: DETR extends naturally to panoptic with a simple mask head, and the instance-level queries make thing-stuff separation elegant

For general-purpose detection, especially with small objects, I’d still reach for YOLOv8 or RT-DETR (which inherits DETR’s query mechanism but fixes the convergence issue). If you’re spending nights debugging NMS edge cases though—and I’ve been there—DETR’s clean architecture might be worth the training cost.

If you’re planning long training runs, a mechanical keyboard with good tactile feedback makes the wait more bearable. Those 500 epochs hit different when your spacebar feels right.

Implementation Details That’ll Trip You Up

A few gotchas from the official implementation:

1. Dropout placement: Dropout is applied after attention in both encoder and decoder, but NOT after the feedforward layers. The paper doesn’t mention this explicitly.

2. Auxiliary losses: DETR adds intermediate decoder layer outputs to the loss. Without this, training doesn’t converge. The paper mentions it briefly, but it’s actually essential—AP drops 4+ points without auxiliary losses.

3. Weight initialization: The classification head’s bias is initialized to $-\log((1-\pi)/\pi)$ where $\pi=0.01$, following focal loss conventions. This prior pushes initial predictions toward “no object,” which helps with the class imbalance (100 queries, usually <10 objects).

# Classification head initialization from official DETR
prior_prob = 0.01
bias_value = -math.log((1 - prior_prob) / prior_prob)
nn.init.constant_(self.class_embed.bias, bias_value)

4. Learning rate schedule: The backbone uses 1/10th the learning rate of the Transformer. I’ve seen people miss this and wonder why their runs diverge.

FAQ

Q: Does DETR require more GPU memory than Faster R-CNN?

DETR’s memory footprint is actually comparable to Faster R-CNN at the same backbone and resolution. The 100 queries × 850 spatial positions attention isn’t that expensive compared to RPN’s dense anchor computations. The DC5 variant is the exception—dilated convolutions at high resolution can easily push past 24GB.

Q: Can DETR be used for real-time detection?

Not the original DETR. At 28 FPS on a V100 (versus YOLO’s 100+ FPS), it’s too slow for real-time applications. RT-DETR and YOLO-World have since achieved real-time speeds with similar architectures, but they required significant architectural modifications beyond the original paper.

Q: Why does DETR always output exactly 100 predictions?

The 100 queries are architectural, not data-dependent. All 100 always produce predictions, but unmatched queries are trained to predict “no object” (∅). At inference, you threshold by confidence—typically keeping predictions above 0.7. The fixed count simplifies batching and removes the need for dynamic memory allocation.

References

• Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020). End-to-End Object Detection with Transformers. ECCV 2020. https://arxiv.org/abs/2005.12872
• Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. NeurIPS 2015. https://arxiv.org/abs/1506.01497
• Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2021). Deformable DETR: Deformable Transformers for End-to-End Object Detection. ICLR 2021. https://arxiv.org/abs/2010.04159
• Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS 2017. https://arxiv.org/abs/1706.03762

DETR proved Transformers could match CNNs at detection without the Frankenstein pipeline. The convergence issue stings, and small object performance needs work. But the conceptual cleanliness of treating detection as set prediction—that insight is already reshaping how we think about spatial reasoning in vision models. I’m watching the DETR family (Deformable, RT-DETR, Co-DETR) closely to see if someone cracks both problems.