DeiT III vs DINOv2: ViT ImageNet Accuracy Without Labels

Two Papers That Changed How We Train Vision Transformers

ViT-Large hitting 87.7% top-1 accuracy on ImageNet without seeing a single label during pretraining. That’s the headline from Meta AI’s DINOv2, and it finally closes a gap that’s been bugging me since the original ViT paper dropped.

But here’s what’s interesting: DeiT III (Touvron et al., 2022) came out a year earlier and achieved 87.2% with a supervised recipe. Same architecture family, nearly identical numbers, completely different training philosophies. Which approach actually wins, and more importantly — which would you deploy?

You can read the DeiT III paper here and DINOv2 here.

The DeiT III Training Recipe: Supervised, But Make It Simple

DeiT III is the third iteration of Data-efficient Image Transformers from Meta. The core insight isn’t a new architecture — it’s a training recipe. The authors stripped away the distillation token that made DeiT I famous and focused purely on getting the fundamentals right.

The recipe boils down to a few key changes from the original ViT training:

• 3-Augment instead of RandAugment: They use only grayscale conversion, solarization, and Gaussian blur. That’s it. No aggressive crops, no color jittering. Sounds almost too simple, right?
• Simple Random Cropping: A random crop with scale (0.08, 1.0) and ratio (3/4, 4/3), then resize to 224×224. They explicitly avoid the complex multi-scale training schemes.
• Binary Cross-Entropy loss: Not standard softmax cross-entropy. This was a surprise. The loss becomes:

$$L_{BCE} = -\frac{1}{C} \sum_{i=1}^{C} [y_i \log(\sigma(z_i)) + (1-y_i) \log(1-\sigma(z_i))]$$

where $\sigma$ is the sigmoid function and $C$ is the number of classes (1000 for ImageNet). This treats each class prediction as an independent binary classification problem.

• LayerScale: A per-channel learnable scaling initialized to a small value $\epsilon = 10^{-6}$. The output of each residual block gets multiplied by a learnable diagonal matrix:

$$x_{l+1} = x_l + \text{diag}(\lambda_l) \cdot \text{Block}_l(x_l)$$

The training runs for 800 epochs with a cosine learning rate schedule, peak lr of $3 \times 10^{-3}$ for ViT-Base, batch size 2048, and AdamW optimizer with weight decay 0.05.

DINOv2: Self-Supervised at Scale

DINOv2 (Oquab et al., 2023) takes a fundamentally different approach. No labels during pretraining at all. The model learns representations by solving a self-supervised objective that combines DINO’s self-distillation with iBOT’s masked image modeling.

The training objective has two components:

$$L = L_{DINO} + L_{iBOT}$$

The DINO loss enforces consistency between global image views through a student-teacher framework with centering and sharpening:

$$L_{DINO} = -\sum_{x \in V_g} p_t(x) \log p_s(x)$$

where $p_t$ and $p_s$ are the softmax-normalized outputs of teacher and student networks, and $V_g$ represents global views. The teacher is an exponential moving average of the student with momentum $\tau$:

$$\theta_t \leftarrow \tau \theta_t + (1-\tau) \theta_s$$

The iBOT component adds masked prediction — the student tries to reconstruct patch tokens that were masked, using the teacher’s output as the target. This combination gives you both global semantic understanding (DINO) and local patch-level features (iBOT).

But the real secret sauce is data. DINOv2 trains on LVD-142M, a curated dataset of 142 million images. They built an automated pipeline to deduplicate, filter unsafe content, and balance the distribution. My best guess is this data curation matters more than any architectural choice.

Why BCE Loss Works Better Than Cross-Entropy for ViT

This is the part that surprised me most when reading DeiT III.

Standard softmax cross-entropy computes:

$$L_{CE} = -\log \frac{e^{z_k}}{\sum_j e^{z_j}}$$

where $k$ is the correct class. This couples all class predictions together — boosting the correct class logit automatically suppresses others.

BCE treats each class independently. The gradient for class $i$ depends only on $z_i$ and whether $y_i = 1$:

$$\frac{\partial L_{BCE}}{\partial z_i} = \sigma(z_i) – y_i$$

The authors report this improves both training stability and final accuracy, especially for larger models. ViT-H with BCE reaches 86.7% while the same architecture with cross-entropy plateaus around 85.8%. I haven’t fully worked out why this helps ViTs specifically but not CNNs — the paper suggests it relates to attention pattern stability, though they don’t prove it rigorously.

ImageNet Accuracy: The Numbers That Matter

Note the “linear” qualifier for DINOv2 — that’s just training a linear classifier on frozen features. If you finetune the full model, DINOv2-g hits 87.7%. Same as DeiT III-H, but DINOv2 required no labeled data during the expensive pretraining phase.

The catch? LVD-142M is proprietary. You can’t reproduce DINOv2’s pretraining without either building your own massive curated dataset or using their released checkpoints.

What Actually Transfers to Downstream Tasks

ImageNet accuracy is table stakes. What about dense prediction, semantic segmentation, depth estimation?

DINOv2 dominates here. On ADE20K semantic segmentation with a linear probe, DINOv2-g achieves 49.0 mIoU versus 43.2 for supervised ViT-L trained on ImageNet-21k. That’s a substantial gap.

Why? Self-supervised learning with masked reconstruction forces the model to learn about spatial relationships and local structures. Supervised classification just needs to learn “this blob of pixels means golden retriever.” Dense tasks reward the richer representations.

Here’s a rough comparison of what you can extract:

import torch
from torchvision import transforms

# DINOv2 gives you genuinely useful intermediate features
dinov2 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitl14')
dinov2.eval()

# Get patch tokens — these are the dense features
with torch.no_grad():
    # x is (B, 3, 518, 518) — note the 518, not 224
    # DINOv2 was trained at 518x518, and this matters
    features = dinov2.forward_features(x)
    patch_tokens = features['x_norm_patchtokens']  # (B, N, D)
    cls_token = features['x_norm_clstoken']  # (B, D)

# patch_tokens.shape: (batch, 37*37, 1024) for ViT-L with patch_size=14
# That's 1369 spatial positions with 1024-dim features each
print(f"Patch tokens shape: {patch_tokens.shape}")

One thing that tripped me up: DINOv2 ViT-L expects 518×518 input, not 224. The model interpolates position embeddings at inference if you use different resolutions, but performance drops noticeably at 224.

The Ablation That Changes Everything

In DeiT III, Table 3 shows what happens when you remove components from their recipe:

LayerScale contributes 0.7% alone. That’s surprisingly high for what’s essentially a learnable scaling factor. But here’s what I find most interesting: using RandAugment instead of their simpler 3-Augment actually hurts. More augmentation isn’t better.

For DINOv2, the key ablation is the data curation. When they train on uncurated data (just scraping images without their filtering pipeline), performance drops by 3-4% across the board. Data quality beats data quantity, even at 142M scale. I’m not entirely sure their filtering criteria generalize to other domains, though — they optimize for ImageNet-like visual concepts.

Implementation Gotchas for Practitioners

If you’re actually going to use these models, here’s what the papers don’t emphasize enough:

DeiT III: The learning rate warmup is crucial. They use 5 epochs of linear warmup to peak lr, and if you skip this or shorten it significantly, training destabilizes around epoch 50-100. I’ve seen people try 1 epoch warmup because “it works for ResNets” and then wonder why loss explodes.

Also, the batch size of 2048 isn’t arbitrary. Smaller batches need different learning rates — roughly $\text{lr} \propto \sqrt{\text{batch\_size}}$. The paper uses linear scaling but sqrt seems more stable empirically:

$$\text{lr}_{new} = \text{lr}_{base} \times \sqrt{\frac{\text{batch}_{new}}{\text{batch}_{base}}}$$

DINOv2: The teacher EMA momentum schedule is sensitive. They use a cosine schedule from 0.994 to 1.0 over training. Start with momentum too high and the teacher never adapts; too low and the student-teacher collapse into identical representations.

# Momentum schedule from DINOv2
import numpy as np

def get_momentum_schedule(base_momentum=0.994, final_momentum=1.0, num_iters=100000):
    momentum = final_momentum - (final_momentum - base_momentum) * (
        np.cos(np.pi * np.arange(num_iters) / num_iters) + 1
    ) / 2
    return momentum

# At iteration 0: momentum = 0.994
# At iteration 50000: momentum ≈ 0.997
# At iteration 100000: momentum = 1.0

The centering operation in DINO loss also deserves attention. Without it, the model converges to a trivial solution where all outputs collapse to the same vector. The center is updated as:

$$c \leftarrow mc + (1-m) \bar{p_t}$$

with $m = 0.9$. This running mean prevents mode collapse but introduces another hyperparameter you need to tune if you’re working at different scales.

When to Use Which

Pick DeiT III if:
– You have labeled data and want maximum sample efficiency
– Training compute is limited (supervised is still cheaper per-accuracy-point)
– You need to train from scratch on a custom domain where labels exist
– Your task is pure classification and you don’t need dense features

Pick DINOv2 if:
– You’re doing dense prediction (segmentation, depth, correspondence)
– You want to use off-the-shelf features without finetuning
– Your downstream task has limited labeled data
– You need features that transfer across visual domains

For most practitioners, I’d start with DINOv2 pretrained weights. The features are ridiculously good out of the box, and you save weeks of pretraining compute. If you need better classification accuracy and have sufficient labels, then finetune with the DeiT III recipe.

BTW, debugging vision model training at 3am is significantly improved by Blue Light Blocking Glasses — your eyes will thank you after staring at loss curves all night.

Limitations Worth Acknowledging

DeiT III doesn’t work as well on datasets with long-tailed class distributions. BCE loss treats all classes equally, which sounds good until you have 90% of samples in 10 classes and 10% spread across 990 classes. The authors acknowledge this but don’t provide a solution.

DINOv2’s main limitation is reproducibility. You can’t train it yourself without massive compute and access to similar training data. Meta released the model weights but not the data pipeline in full detail. For a paper about “learning without labels,” the labels ended up mattering indirectly through the data curation process — they used ImageNet categories as concepts for retrieval in building LVD-142M.

And both papers evaluate primarily on ImageNet-like domains. Would these results hold for medical imaging, satellite imagery, or microscopy? The DeiT III recipe probably transfers, but DINOv2’s data distribution is heavily biased toward web-scraped natural images.

FAQ

Q: Can I use DINOv2 features without any finetuning?
Yes, and this is actually a strength. For many downstream tasks, training just a linear classifier on frozen DINOv2 features gives competitive results. This is called linear probing, and DINOv2-g achieves 87.0% ImageNet top-1 with just a learned linear layer on top.

Q: Why does DeiT III use BCE instead of cross-entropy loss?
BCE decouples gradients between classes, which appears to improve training stability for ViTs specifically. Each class gets updated based only on whether it was the target, without the coupled normalization of softmax. The authors report ~0.4% accuracy gain from this change alone.

Q: How much compute does it take to train these models?
DeiT III-L on ImageNet-21k uses 800 A100 GPU hours. DINOv2-g on LVD-142M takes approximately 22,000 A100 GPU hours for the full 500k iteration training. The self-supervised approach is significantly more expensive but eliminates labeling costs.

References

• Touvron, H., Cord, M., & Jégou, H. (2022). DeiT III: Revenge of the ViT. ECCV 2022. arXiv:2204.07118
• Oquab, M., et al. (2023). DINOv2: Learning Robust Visual Features without Supervision. arXiv preprint. arXiv:2304.07193
• Dosovitskiy, A., et al. (2021). An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale. ICLR 2021. arXiv:2010.11929
• Caron, M., et al. (2021). Emerging Properties in Self-Supervised Vision Transformers (DINO). ICCV 2021. arXiv:2104.14294
• Zhou, J., et al. (2022). iBOT: Image BERT Pre-Training with Online Tokenizer. ICLR 2022. arXiv:2111.07832

Self-supervised learning finally matched supervised accuracy at scale. What I’m curious about next: can we get similar results with 10x less data through better curation? LVD-142M feels like brute force. Someone’s going to crack the efficient version — the CLIP team showed clever data collection beats raw scale, and I suspect the same principle applies here.