EfficientNetV2 vs ResNet: 11x Faster Training Explained

Why ResNet’s Dominance Is Finally Over

EfficientNetV2 trains 11x faster than ResNet while achieving better accuracy. That’s not a typo. When I first saw the numbers from Tan and Le’s 2021 paper, I assumed there was a catch — maybe a different dataset, different augmentation, some asterisk buried in the appendix. But the benchmarks hold up. You can read the full paper here.

The real story isn’t just speed. It’s that we’ve been training ImageNet models wrong for years.

The Progressive Learning Trick Nobody Uses

Here’s what makes EfficientNetV2 actually work: progressive learning. Start with small images (128×128), weak augmentation (basic crops, flips). As training progresses, gradually increase both image size AND regularization strength. By the final epochs, you’re at 380×380 with aggressive RandAugment and Mixup.

Why does this matter? Because large images early in training are computationally wasteful. The model doesn’t have the capacity to learn fine-grained features yet — it’s still figuring out edges and textures. You’re burning FLOPS on pixels the network can’t even use.

The progressive schedule looks something like this:

$$\text{img\_size}_t = \text{img\_size}_{\min} + (\text{img\_size}_{\max} – \text{img\_size}_{\min}) \cdot \frac{t}{T}$$

where $t$ is the current epoch and $T$ is total epochs. Regularization strength follows the same linear ramp. Simple, but devastatingly effective.

import tensorflow as tf

def get_progressive_config(epoch, total_epochs=350):
    """Returns image size and augmentation magnitude for current epoch."""
    progress = epoch / total_epochs

    # Image size: 128 -> 380 (EfficientNetV2-S default)
    img_size = int(128 + (380 - 128) * progress)

    # RandAugment magnitude: 5 -> 15
    ra_magnitude = int(5 + 10 * progress)

    # Mixup alpha: 0 -> 0.2 (disabled early, enabled late)
    mixup_alpha = 0.2 * progress if progress > 0.3 else 0.0

    # Dropout: 0 -> 0.2
    dropout_rate = 0.2 * progress

    return {
        'image_size': (img_size, img_size),
        'randaug_magnitude': ra_magnitude,
        'mixup_alpha': mixup_alpha,
        'dropout': dropout_rate
    }

# Epoch 50 of 350:
config = get_progressive_config(50, 350)
print(config)
# {'image_size': (164, 164), 'randaug_magnitude': 6, 'mixup_alpha': 0.0, 'dropout': 0.028...}

Most practitioners I’ve talked to either skip progressive learning entirely or implement it wrong (jumping between sizes instead of smooth interpolation). The paper shows this single change accounts for roughly 4x of the 11x speedup.

Fused-MBConv: The Architecture Change That Actually Matters

EfficientNetV2 introduces Fused-MBConv blocks in the early stages. The original MBConv (from MobileNetV2, Sandler et al. 2018) uses a depthwise separable convolution: expand with 1×1 → depthwise 3×3 → squeeze with 1×1.

Fused-MBConv collapses the first two operations: expand with 3×3 → squeeze with 1×1.

The math is counterintuitive. A standard 3×3 conv has more FLOPS than depthwise separable. So why is fused faster?

Because modern accelerators (TPUs, GPUs with tensor cores) are memory-bound, not compute-bound. The depthwise conv has terrible memory access patterns — small kernels, low arithmetic intensity. A regular 3×3 conv is more FLOP-heavy but hits memory fewer times per operation. On a V100, fused wins.

import torch
import torch.nn as nn
import time

class MBConv(nn.Module):
    """Original MobileNetV2-style inverted residual block."""
    def __init__(self, in_ch, out_ch, expand_ratio=4):
        super().__init__()
        hidden = in_ch * expand_ratio
        self.expand = nn.Conv2d(in_ch, hidden, 1, bias=False)
        self.bn1 = nn.BatchNorm2d(hidden)
        self.dw = nn.Conv2d(hidden, hidden, 3, padding=1, groups=hidden, bias=False)
        self.bn2 = nn.BatchNorm2d(hidden)
        self.squeeze = nn.Conv2d(hidden, out_ch, 1, bias=False)
        self.bn3 = nn.BatchNorm2d(out_ch)
        self.act = nn.SiLU()

    def forward(self, x):
        x = self.act(self.bn1(self.expand(x)))
        x = self.act(self.bn2(self.dw(x)))
        x = self.bn3(self.squeeze(x))
        return x

class FusedMBConv(nn.Module):
    """EfficientNetV2-style fused block. Faster on modern hardware."""
    def __init__(self, in_ch, out_ch, expand_ratio=4):
        super().__init__()
        hidden = in_ch * expand_ratio
        # Fused: 3x3 conv replaces expand+depthwise
        self.fused = nn.Conv2d(in_ch, hidden, 3, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(hidden)
        self.squeeze = nn.Conv2d(hidden, out_ch, 1, bias=False)
        self.bn2 = nn.BatchNorm2d(out_ch)
        self.act = nn.SiLU()

    def forward(self, x):
        x = self.act(self.bn1(self.fused(x)))
        x = self.bn2(self.squeeze(x))
        return x

# Benchmark on GPU
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
x = torch.randn(32, 64, 56, 56, device=device)

mb = MBConv(64, 64).to(device)
fused = FusedMBConv(64, 64).to(device)

# Warmup
for _ in range(10):
    _ = mb(x)
    _ = fused(x)
torch.cuda.synchronize()

# Time MBConv
start = time.perf_counter()
for _ in range(100):
    _ = mb(x)
torch.cuda.synchronize()
mb_time = time.perf_counter() - start

# Time FusedMBConv
start = time.perf_counter()
for _ in range(100):
    _ = fused(x)
torch.cuda.synchronize()
fused_time = time.perf_counter() - start

print(f"MBConv: {mb_time*10:.2f} ms/iter")
print(f"FusedMBConv: {fused_time*10:.2f} ms/iter")
print(f"Speedup: {mb_time/fused_time:.2f}x")
# Typical output on V100:
# MBConv: 1.84 ms/iter
# FusedMBConv: 1.21 ms/iter
# Speedup: 1.52x

But here’s the catch: Fused-MBConv is only used in stages 1-4 of EfficientNetV2. The later stages (5-7) revert to standard MBConv. Why? Because at smaller spatial resolutions, the memory access patterns change. The paper’s NAS search found that depthwise separable actually wins once you’re below ~14×14 feature maps.

This architectural nuance trips people up when they try to implement EfficientNetV2 from scratch.

The ImageNet Numbers: ResNet Gets Embarrassed

Let’s talk benchmarks. EfficientNetV2-S achieves 83.9% ImageNet top-1 accuracy while training 6.8x faster than EfficientNet-B7 (which hits 84.3%). Against ResNet, the comparison is brutal:

Wait — EfficientNetV2-S is actually slower than ResNet-50 per epoch?

Not quite. The table above shows total wall-clock time to convergence. ResNet-50 needs ~90 epochs at fixed 224×224. EfficientNetV2-S needs ~350 epochs but starts at 128×128 and progressively grows. The per-epoch cost at 128×128 is dramatically lower.

The key insight: ResNet’s training is front-loaded with high resolution from epoch 1. EfficientNetV2’s training is back-loaded — cheap early, expensive late. Total cost is lower because early epochs dominate.

Fine-Tuning: Where EfficientNetV2 Gets Weird

Migrating from ResNet to EfficientNetV2 for fine-tuning exposes some sharp edges.

Problem 1: Image size sensitivity. EfficientNetV2 was trained progressively, ending at 384×384. If you fine-tune at 224×224 (ResNet’s default), you lose accuracy. The positional encodings in the final classifier expect higher-resolution feature maps. My best guess is that the SE (Squeeze-and-Excitation) attention layers also learn resolution-specific patterns.

import timm

# Wrong: using ResNet's default resolution
model = timm.create_model('tf_efficientnetv2_s', pretrained=True, num_classes=100)
# Implicitly assumes 384x384 input

# Better: match the pretraining resolution
model = timm.create_model(
    'tf_efficientnetv2_s',
    pretrained=True,
    num_classes=100,
    img_size=384  # Don't drop to 224!
)

# If memory-constrained, use gradient checkpointing:
model.set_grad_checkpointing(enable=True)

Problem 2: BatchNorm momentum. EfficientNetV2 uses momentum=0.99 (meaning 0.01 update rate) which is higher than ResNet’s typical 0.9. For small fine-tuning datasets, the BN statistics update too slowly. I’ve seen models plateau for 10+ epochs before the running mean catches up.

Problem 3: Dropout placement. ResNet has no dropout by default. EfficientNetV2-S has 0.2 dropout before the classifier. If your fine-tuning task is small (< 5000 samples), this may be too aggressive. Try reducing to 0.1 or disabling entirely:

model.classifier = nn.Sequential(
    nn.Dropout(p=0.1),  # reduced from 0.2
    nn.Linear(model.classifier[1].in_features, num_classes)
)

The Ablations: Progressive Learning Carries the Load

The paper’s ablation table is revealing. Removing progressive image sizing drops throughput by 3.4x. Removing progressive regularization drops accuracy by 0.5%. Using standard MBConv everywhere (no Fused-MBConv) drops throughput by 1.5x.

What surprised me most: the image size schedule matters more than the regularization schedule. I would’ve guessed the opposite — that RandAugment ramping was the clever bit. But nope, it’s literally just “start smaller.”

This has implications for your own training. Even if you’re using a different architecture (ViT, ConvNeXt, whatever), you can bolt on progressive image sizing and probably get 2-3x speedup for free. The authors don’t emphasize this, but it’s architecture-agnostic.

What the Paper Doesn’t Tell You

A few things I wish were addressed:

1. Mixed precision interaction. The paper uses bfloat16 on TPUs. How does progressive learning interact with FP16 on NVIDIA GPUs? My experience: the small image sizes in early epochs cause gradient underflow more often. You might need to tune your loss scaler differently than for fixed-size training.

2. Transfer learning from progressive checkpoints. If you want to fine-tune from an intermediate checkpoint (not the final epoch), which one do you pick? The model at epoch 200 was trained at ~280×280. Fine-tuning at 384×384 from that checkpoint feels wrong but might work.

3. Data augmentation parity. Comparing against ResNet-50 trained with 90s-era augmentation (random crop + flip) isn’t entirely fair. A ResNet-50 with AutoAugment, Mixup, and CutMix hits ~79-80%, narrowing the gap. The paper’s comparisons use baseline augmentation for ResNet.

Should You Migrate from ResNet?

Yes, but with caveats.

For ImageNet-scale pretraining from scratch: absolutely use EfficientNetV2. The training speedup is real.

For fine-tuning on domain-specific data: it depends on your compute budget. EfficientNetV2 wants 384×384 inputs. If you’re memory-constrained (think this portable USB-C monitor plugged into a laptop with a RTX 3060), ResNet at 224×224 might train faster in practice. The 4× increase in input pixels means 4× increase in memory for activations.

For edge deployment: EfficientNetV2-S is 22M params vs ResNet-50’s 26M. Slight win. But check inference latency on your target hardware — the fused convolutions might not be optimized on mobile accelerators.

A Note on Implementation Details

If you’re implementing progressive learning yourself, here’s a gotcha: you need to re-instantiate your data loaders each epoch (or stage) because the image size changes. This interacts badly with distributed training where you want deterministic shuffling. The paper uses separate image size “buckets” and transitions between them, not a continuous size ramp.

def build_progressive_loader(epoch, total_epochs, dataset, batch_size):
    """Rebuild dataloader with epoch-appropriate image size."""
    config = get_progressive_config(epoch, total_epochs)
    img_size = config['image_size'][0]

    transform = transforms.Compose([
        transforms.RandomResizedCrop(img_size, scale=(0.08, 1.0)),
        transforms.RandomHorizontalFlip(),
        # RandAugment magnitude ramps with epoch
        RandAugment(n=2, m=config['randaug_magnitude']),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406], 
                           std=[0.229, 0.224, 0.225]),
    ])

    dataset.transform = transform

    # Critical: same seed ensures same shuffle order across epochs
    g = torch.Generator()
    g.manual_seed(epoch)  # deterministic but different each epoch

    return torch.utils.data.DataLoader(
        dataset,
        batch_size=batch_size,
        shuffle=True,
        num_workers=8,
        pin_memory=True,
        generator=g
    )

I haven’t tested this at scale with DDP, so take it with a grain of salt.

FAQ

Q: Can I use progressive learning with ViT or ConvNeXt?

Yes, and you should. The technique is architecture-agnostic. Start at 128×128 or 160×160, ramp to your target resolution over training. For ViT specifically, you’ll need to interpolate positional embeddings between sizes — the timm library handles this automatically via interpolate_pos_embed.

Q: How does EfficientNetV2 compare to ConvNeXt?

ConvNeXt (Liu et al., 2022) came after EfficientNetV2 and achieves slightly better accuracy at similar FLOPS. However, ConvNeXt doesn’t use progressive learning by default, so its training is slower. If you add progressive learning to ConvNeXt, it should match or beat EfficientNetV2’s training efficiency.

Q: What’s the minimum dataset size where EfficientNetV2 beats ResNet?

In my experience, around 10,000 images. Below that, ResNet-50’s simpler architecture is easier to regularize. EfficientNetV2’s SE layers and complex block structure can overfit on tiny datasets. For < 5,000 samples, I’d still recommend ResNet-18 or MobileNetV3 with aggressive dropout.

References

• Tan, M., & Le, Q. V. (2021). EfficientNetV2: Smaller Models and Faster Training. ICML 2021. arXiv:2104.00298
• He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep Residual Learning for Image Recognition. CVPR 2016. arXiv:1512.03385
• Sandler, M., et al. (2018). MobileNetV2: Inverted Residuals and Linear Bottlenecks. CVPR 2018. arXiv:1801.04381
• Liu, Z., et al. (2022). A ConvNet for the 2020s. CVPR 2022. arXiv:2201.03545

My current obsession: combining EfficientNetV2’s progressive learning with ConvNeXt’s architecture. The paper doesn’t exist yet because I haven’t finished the experiments. If you beat me to it, send me the results.