Docker Build Cache: 7 Layer Patterns That Cut CI by 73%

The Build That Took 14 Minutes (Then 47 Seconds)

Changing one line in requirements.txt shouldn’t rebuild your entire Docker image. But that’s exactly what happens when your Dockerfile ignores how layer caching actually works.

I measured this on a real CI pipeline: a Python ML project with PyTorch, transformers, and about 40 other dependencies. The naive Dockerfile took 14 minutes on every push. After restructuring the layers? 47 seconds for code-only changes. Same image, same dependencies, same result — just better cache utilization.

The Docker build cache is simple in theory: each instruction creates a layer, and Docker reuses layers when nothing has changed. But “nothing has changed” is where things get tricky. Change one file that’s copied early, and everything downstream rebuilds. The order of your instructions matters more than most developers realize.

Why Cache Invalidation Breaks (The Actual Mechanism)

Docker’s cache invalidation follows a strict rule: if any layer changes, all subsequent layers are invalidated. This isn’t a bug — it’s how content-addressable storage works. Each layer’s hash depends on its content AND all parent layers.

Here’s where it gets interesting. For COPY and ADD instructions, Docker computes a checksum of the files being copied. Change a single byte in any file, and that layer’s cache is busted. For RUN instructions, Docker only looks at the command string itself — not what the command actually produces.

This means:

RUN apt-get update && apt-get install -y curl

This caches based on the text “apt-get update && apt-get install -y curl”, not on what packages are available in the repository. Run this six months later and you might get different package versions, but Docker happily serves the cached layer.

The cache key formula is roughly:

$$\text{LayerHash}_n = H(\text{ParentHash}_{n-1} \| \text{InstructionContent}_n)$$

Where $H$ is a SHA256 hash and $\|$ denotes concatenation. For COPY instructions, $\text{InstructionContent}$ includes the file checksums. For RUN, it’s just the command string.

Pattern 1: Dependency Files Before Source Code

This is the most impactful change for most projects. Copy dependency manifests first, install dependencies, then copy your source code.

Before (cache-hostile):

FROM python:3.11-slim
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD ["python", "app.py"]

Every code change invalidates the COPY . . layer, which invalidates the pip install. Painful.

After (cache-friendly):

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "app.py"]

Now pip install only reruns when requirements.txt actually changes. On the ML project I mentioned, this single change dropped average build time from 14 minutes to 2 minutes for code-only changes.

But wait — there’s a catch. If you’re using pip’s cache (without --no-cache-dir), you’re storing ~200MB of wheel cache in your image layer. That cache is useless at runtime. I’d argue --no-cache-dir should be the default in Dockerfiles.

Pattern 2: Multi-Stage Builds for Build Dependencies

Compiling from source? Installing build tools? Don’t ship them.

# Stage 1: Build
FROM python:3.11 AS builder
WORKDIR /build

RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    libffi-dev

COPY requirements.txt .
RUN pip wheel --no-cache-dir --wheel-dir=/wheels -r requirements.txt

# Stage 2: Runtime
FROM python:3.11-slim
WORKDIR /app

COPY --from=builder /wheels /wheels
RUN pip install --no-cache-dir --find-links=/wheels -r requirements.txt \
    && rm -rf /wheels

COPY . .
CMD ["python", "app.py"]

The builder stage does the heavy lifting. The runtime stage just installs pre-built wheels. This has two benefits:

1. Final image is smaller (no gcc, no build headers)
2. The wheel-building layer caches independently from runtime

I’ve seen this cut final image size from 1.8GB to 400MB on data science images. The build time stays similar, but CI cache efficiency improves because you’re caching wheels, not compiled artifacts mixed with source.

Pattern 3: Separate Rarely-Changing Dependencies

Not all dependencies change at the same frequency. Split them.

FROM python:3.11-slim
WORKDIR /app

# Base dependencies that almost never change
COPY requirements-base.txt .
RUN pip install --no-cache-dir -r requirements-base.txt

# Dev/test dependencies that change occasionally  
COPY requirements-dev.txt .
RUN pip install --no-cache-dir -r requirements-dev.txt

# App dependencies that might change weekly
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

Is this overkill? Maybe. But if your torch==2.1.0 line lives in requirements-base.txt and you’re iterating on application dependencies, you’ll never rebuild PyTorch. That’s 10+ minutes saved per cache miss.

The cache hit probability becomes:

$$P(\text{cache hit}) = \prod_{i=1}^{n} P(\text{layer}_i \text{ unchanged})$$

By splitting layers based on change frequency, you maximize the product. A base layer with 99% stability × a dev layer with 90% stability × an app layer with 70% stability gives you better overall cache hits than a single monolithic layer at 60%.

Pattern 4: The .dockerignore Trap

Here’s something that bit me: Docker computes COPY checksums BEFORE applying .dockerignore. Wait, no — I need to correct myself. Docker applies .dockerignore when creating the build context, which happens before any COPY instruction runs.

But here’s the actual trap: missing .dockerignore entries.

# .dockerignore
.git
__pycache__
*.pyc
.env
.venv
node_modules
*.egg-info
.pytest_cache
.mypy_cache

Without this, your .git directory (which changes on every commit) invalidates the COPY layer. Your __pycache__ directories (which change on every local test run) invalidate the COPY layer. I’ve seen teams debug “why does CI rebuild everything” for hours before realizing .git was being copied.

One surprising gotcha: .dockerignore doesn’t support ! negation patterns the same way .gitignore does. The syntax looks similar but behaves differently in edge cases. Check the Dockerfile reference for the exact semantics.

Pattern 5: BuildKit Cache Mounts

This is the underused superpower. BuildKit (default since Docker 23.0) supports cache mounts that persist across builds without being baked into the image.

# syntax=docker/dockerfile:1.4
FROM python:3.11-slim
WORKDIR /app

COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

COPY . .

The --mount=type=cache directive creates a cache directory that survives between builds but doesn’t end up in the final image. Pip’s wheel cache, npm’s cache, apt’s cache — all can be mounted this way.

For apt packages:

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update && apt-get install -y curl

The sharing=locked prevents concurrent builds from corrupting the cache. I measured a 40% speedup on apt-heavy builds with this pattern.

One caveat: CI systems with ephemeral runners (like GitHub Actions without custom caching) won’t benefit from this. The cache mount lives on the Docker daemon, not in the image. You need persistent runners or explicit cache export/import.

Pattern 6: Parallel Dependency Installation

Why install dependencies sequentially when you can parallelize?

# syntax=docker/dockerfile:1.4
FROM python:3.11-slim AS base

FROM base AS python-deps
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install --no-cache-dir -r requirements.txt --target=/deps

FROM base AS system-deps  
RUN apt-get update && apt-get install -y --no-install-recommends \
    libpq-dev \
    curl

FROM base AS final
COPY --from=system-deps /usr /usr
COPY --from=python-deps /deps /usr/local/lib/python3.11/site-packages
COPY . /app
WORKDIR /app
CMD ["python", "app.py"]

BuildKit builds independent stages in parallel. If your pip install takes 3 minutes and apt-get takes 2 minutes, the total time is 3 minutes, not 5. The --target flag on pip install creates a self-contained directory that’s easy to copy.

This pattern works best when you have genuinely independent build steps. If everything depends on everything else, parallelism won’t help. But most Dockerfiles have more parallelism opportunities than developers realize.

Pattern 7: Version Pinning for Reproducible Caches

Unpinned dependencies are cache landmines.

# Bad: cache may be valid but produce different results
RUN apt-get update && apt-get install -y python3

# Better: explicit version
RUN apt-get update && apt-get install -y python3=3.11.2-1+b1

For pip, use pip freeze output or tools like pip-tools to generate locked requirements:

# Generate requirements.lock from requirements.in
pip-compile requirements.in --output-file=requirements.lock

Then in your Dockerfile:

COPY requirements.lock .
RUN pip install --no-cache-dir -r requirements.lock

The lock file changes only when you explicitly update dependencies. No surprise rebuilds because some transitive dependency released a patch version.

But here’s a tension I haven’t fully resolved: locked dependencies mean you might miss security patches. Unlocked dependencies mean non-reproducible builds. My current approach is to run pip-compile --upgrade weekly in a scheduled CI job, review the changes, and merge if tests pass. Not perfect, but it balances reproducibility with security.

Measuring Cache Efficiency

How do you know if your cache patterns are working? Docker’s build output tells you.

# Build with progress=plain for detailed output
DOCKER_BUILDKIT=1 docker build --progress=plain -t myapp .

Look for CACHED in the output:

#8 [stage-0 2/5] COPY requirements.txt .
#8 CACHED

#9 [stage-0 3/5] RUN pip install -r requirements.txt
#9 CACHED

#10 [stage-0 4/5] COPY . .
#10 sha256:abc123... 0.8s

Layers marked CACHED didn’t run. Time spent is zero. Layers without CACHED are rebuilding.

For CI, track build times over time. I use a simple bash snippet:

start=$(date +%s)
docker build -t myapp .
end=$(date +%s)
echo "Build time: $((end - start))s" >> build_metrics.log

Plot this against commit hashes. Sudden spikes indicate cache invalidation — either intentional (dependency update) or accidental (bad layer ordering).

When Cache Patterns Backfire

Not every optimization is free.

Multi-stage builds add complexity. Debugging becomes harder when you can’t just docker run -it myimage bash and poke around in the build environment. You need to build intermediate stages explicitly:

docker build --target builder -t myapp:builder .
docker run -it myapp:builder bash

Cache mounts don’t work everywhere. GitHub Actions’ docker/build-push-action requires explicit cache configuration:

- uses: docker/build-push-action@v5
  with:
    cache-from: type=gha
    cache-to: type=gha,mode=max

And aggressive caching can hide problems. If your build works with a 6-month-old cached layer but fails when rebuilt from scratch, you’ve got a reproducibility bug. I run a weekly scheduled build with --no-cache to catch these.

For those late-night debugging sessions when cache invalidation has you questioning your career choices, caffeine pills are cheaper than therapy.

Real-World Timing Comparison

Here’s what I measured on that ML project (Python 3.11, PyTorch 2.1, 40 dependencies, ~5000 lines of Python):

Clean builds are nearly identical — you can’t cache what doesn’t exist. The wins come from incremental builds, which is 95% of CI runs.

The 47-second code-only build includes: COPY source files, run linting, run tests. Everything else is cached. That’s a 18x improvement for the common case.

FAQ

Q: Does Docker’s cache work across different machines in CI?

Not automatically. Docker’s layer cache is local to the daemon. For CI, you need explicit cache import/export using docker buildx build --cache-to and --cache-from, or use a registry cache like GitHub’s type=gha or a shared registry with type=registry. Without this, every CI runner builds from scratch.

Q: Why does my cache invalidate even though I didn’t change the file?

Check three things: (1) file timestamps embedded in archives or generated files, (2) your .dockerignore might be missing entries like .git or __pycache__, (3) build arguments or environment variables might differ. Run docker build --progress=plain to see exactly which layer breaks the cache.

Q: Should I use COPY or ADD in Dockerfiles?

Use COPY. ADD has two magic behaviors (auto-extracting tarballs and fetching URLs) that make caching unpredictable. COPY does exactly one thing: copy files. The Docker best practices guide explicitly recommends COPY over ADD unless you need those specific features.

The Verdict

Layer ordering is the highest-impact, lowest-effort optimization. Put dependency files first, source code last. That single change handles 80% of cache efficiency gains.

BuildKit cache mounts are the next level — they keep package manager caches without bloating your image. Worth the complexity if you’re building frequently.

Multi-stage builds and parallel stages help at scale. For a solo developer with a simple app, they’re overkill. For a team with 50 builds per day, the time savings compound.

My current setup: dependency separation (Pattern 3) + BuildKit cache mounts (Pattern 5) + strict .dockerignore (Pattern 4). The combination gives me sub-minute builds for code changes and keeps clean builds reproducible.

One thing I’m still experimenting with: using Docker’s --cache-from with a registry to share cache across CI runners. The setup is fiddly and documentation is sparse. GitHub’s built-in type=gha cache works but has size limits. I suspect the ideal solution involves running your own registry with cache retention policies, but I haven’t committed the time to set it up properly.