pip install Works, Production Breaks: Dependency Hell Fix

The Package That Wasn’t There

Your CI pipeline is green. pip install ran without errors. The Docker build succeeded. You deploy to production, and suddenly:

ModuleNotFoundError: No module named 'cryptography'

But cryptography is right there in requirements.txt. You can see it in the build logs. It installed successfully. What’s happening?

This isn’t a random failure. It’s the gap between “pip said yes” and “the runtime can actually import it.” I’ve seen this kill three production deployments in one week, each for a different reason.

Why pip install Success Means Nothing

When pip install cryptography exits with code 0, it means one thing: pip successfully placed files on disk. It does NOT mean:

• The package can be imported
• Its binary dependencies are present
• The Python version can load it
• Another installed package won’t break it at runtime

pip operates in “optimistic mode.” It resolves dependencies by reading metadata files (METADATA, PKG-INFO) and picking versions that satisfy declared constraints. But metadata can lie. Or more accurately, metadata can’t capture the full runtime reality.

Here’s what actually happens during resolution:

$$\text{Installable} = \{p \mid \forall d \in \text{deps}(p), \exists v : \text{satisfies}(d, v)\}$$

This set-theoretic view says “a package is installable if for every dependency $d$, there exists some version $v$ that satisfies the constraint.” But the real compatibility check is:

$$\text{Runnable} = \{p \mid \forall d \in \text{deps}(p), \text{importable}(d) \land \text{compatible}(p, d)\}$$

That importable and compatible predicate? pip never evaluates it. It can’t — it would require running code in every possible combination.

The Three Gaps

Binary Incompatibility (The Wheel Lie)

You’re on Python 3.11, Ubuntu 22.04, x86_64. You run:

pip install onnxruntime-gpu==1.14.0

Succeeds. You check:

import onnxruntime
print(onnxruntime.get_device())

Output:

Segmentation fault (core dumped)

What happened? The wheel was built against CUDA 11.8, but your production container has CUDA 12.1. The .so files loaded, then immediately crashed when they tried to call CUDA symbols that don’t exist in the newer version.

pip downloaded onnxruntime_gpu-1.14.0-cp311-cp311-manylinux_2_17_x86_64.whl. The filename says “any glibc ≥ 2.17, Python 3.11.” It doesn’t say “CUDA 11.8 only.” That constraint lives in the compiled binary, not the metadata.

Transitive Constraint Conflicts (The Backtracking Trap)

Consider this requirements.txt:

requests==2.28.1
urllib3==2.0.0

pip installs both. No errors. But then:

import requests
requests.get('https://google.com')

Raises:

ImportError: cannot import name 'appengine' from 'urllib3.contrib'

The issue: requests 2.28.1 declares urllib3>=1.21.1,<1.27. pip saw urllib3==2.0.0 and thought “2.0.0 is not less than 1.27, but the user explicitly asked for it, so I’ll install it anyway.” This is pip’s “user overrides dependencies” rule.

But at runtime, requests imports urllib3.contrib.appengine, which was removed in urllib3 2.0. pip never checked that import — it only checked version constraints in metadata.

The actual dependency graph:

requests 2.28.1
  └─ urllib3 >=1.21.1,<1.27 (declared)
       └─ urllib3.contrib.appengine (runtime requirement, undeclared)

pip resolves the first level. Python crashes on the second.

Platform Tag Mismatches (The macOS Trap)

You develop on an M1 Mac. You deploy to AWS Lambda (x86_64 Linux). Your local install:

pip install numpy==1.24.0

Downloads numpy-1.24.0-cp311-cp311-macosx_11_0_arm64.whl. Works perfectly. You freeze:

pip freeze > requirements.txt

Production install on Lambda:

pip install -r requirements.txt

Succeeds — but downloads numpy-1.24.0-cp311-cp311-manylinux_2_17_x86_64.whl, a different wheel. This one was built with a different BLAS backend. Your code that relied on specific MKL functions:

import numpy as np
np.show_config()  # Assumes MKL

Fails because the Linux wheel uses OpenBLAS.

Same package name. Same version. Different binary. pip install succeeded both times.

What Production Actually Needs

Production doesn’t care if pip succeeded. It cares about three things:

1. Can Python import the module? Not “did pip place files,” but “does import X work?”
2. Are transitive runtime dependencies present? Not just declared in metadata, but actually loadable.
3. Do binary ABIs match? glibc version, CUDA version, CPU instruction sets.

Here’s a simple runtime check I add to every deployment script:

# verify_imports.py
import sys
import importlib

CRITICAL_IMPORTS = [
    'requests',
    'numpy',
    'sqlalchemy',
    'cryptography',
    # Add your critical dependencies
]

def verify_import(module_name):
    try:
        mod = importlib.import_module(module_name)
        # Try to access a known attribute to ensure the module is fully loaded
        if hasattr(mod, '__version__'):
            print(f"✓ {module_name} {mod.__version__}")
        else:
            print(f"✓ {module_name} (no version attr)")
        return True
    except Exception as e:
        print(f"✗ {module_name}: {e}", file=sys.stderr)
        return False

if __name__ == '__main__':
    results = [verify_import(m) for m in CRITICAL_IMPORTS]
    if not all(results):
        sys.exit(1)

Run this after pip install in your Dockerfile:

RUN pip install -r requirements.txt && python verify_imports.py

If the build fails here, you know immediately — not after deployment.

The pip-compile Solution

The root problem: requirements.txt with unpinned or loosely pinned versions allows pip to make different choices across environments. The fix: lock EVERY transitive dependency with exact versions and hashes.

pip-tools does this:

pip install pip-tools

Create requirements.in with your top-level dependencies:

requests
numpy>=1.24

Generate a fully resolved lock file:

pip-compile --generate-hashes --output-file=requirements.txt requirements.in

This produces:

# requirements.txt (generated)
certifi==2022.12.7 \
    --hash=sha256:35824b4c3a97115964b408844d64aa14db1cc518f6562e8d7261699d1350a9e3
charset-normalizer==3.0.1 \
    --hash=sha256:ebea339af930f8ca5d7a699b921106c6e29c617fe9606fa7baa043c1cdae326f
idna==3.4 \
    --hash=sha256:814f528e8dead7d329833b91c5faa87d60bf71824cd12a7530b5526063d02cb4
numpy==1.24.2 \
    --hash=sha256:003a9f530e880cb2cd177cba1af7220b9aa42def9c4afc2a2fc3ee6be7eb2b22
requests==2.28.2 \
    --hash=sha256:64299f4909223da747622c030b781c0d7811e359c37124b4bd368fb8c6518baa
urllib3==1.26.14 \
    --hash=sha256:076907bf8fd355cde77728471316625a4d2f7e713c125f51953bb5b3eecf4f72

Every single transitive dependency, pinned to an exact version, with a SHA256 hash. If ANY file changes — even if the version number stays the same — pip will refuse to install it.

Now your production install:

pip install --require-hashes -r requirements.txt

This guarantees bit-for-bit identical packages across all environments.

The Docker Multi-Stage Build Pattern

Even with locked dependencies, different base images can cause issues. The solution: build and run in identical environments.

# Build stage
FROM python:3.11-slim-bullseye AS builder

WORKDIR /app

# Install build dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    gcc \
    g++ \
    make \
    libffi-dev \
    && rm -rf /var/lib/apt/lists/*

# Copy and install Python dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir --user --require-hashes -r requirements.txt

# Runtime stage
FROM python:3.11-slim-bullseye

WORKDIR /app

# Copy installed packages from builder
COPY --from=builder /root/.local /root/.local

# Verify imports before finalizing
COPY verify_imports.py .
RUN python verify_imports.py

# Copy application code
COPY . .

CMD ["python", "app.py"]

The key: same base image (python:3.11-slim-bullseye) in both stages. Different base images can have different glibc versions, different system libraries, different OpenSSL versions.

When Hash Verification Fails

You run pip install --require-hashes, and:

ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE.
  numpy==1.24.2 (from -r requirements.txt (line 8)):
    Expected sha256:003a9f530e880cb2cd177cba1af7220b9aa42def9c4afc2a2fc3ee6be7eb2b22
    Got      sha256:8d5b8ed5c8d5e814b5e3e5e5a1c8f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5

This means one of three things:

1. PyPI package was updated in-place (rare but happens for security patches)
2. You’re using a different architecture (M1 vs x86_64)
3. A mirror or cache modified the package (corporate proxy, artifactory)

Don’t just regenerate hashes and move on. Investigate. If it’s case 1, check PyPI’s changelog. If it’s case 2, you need platform-specific lock files. If it’s case 3, you have a serious supply chain issue.

Platform-Specific Lock Files

If you develop on macOS and deploy on Linux, you need TWO lock files:

# On macOS (for local dev)
pip-compile --generate-hashes --platform=macosx_11_0_arm64 \
    --output-file=requirements-mac.txt requirements.in

# On Linux (for production, use Docker or CI)
docker run --rm -v $(pwd):/app python:3.11-slim-bullseye \
    bash -c "pip install pip-tools && pip-compile --generate-hashes \
    --output-file=/app/requirements-linux.txt /app/requirements.in"

Your CI/CD uses requirements-linux.txt. Your local setup uses requirements-mac.txt. They’ll have different wheels but identical version numbers.

The importlib.metadata Final Check

Even after all this, I add one more check in the application startup code:

# app.py
import sys
from importlib.metadata import version, PackageNotFoundError

REQUIRED_VERSIONS = {
    'requests': '2.28.2',
    'numpy': '1.24.2',
    'sqlalchemy': '2.0.0',
}

def verify_runtime_versions():
    mismatches = []
    for pkg, expected in REQUIRED_VERSIONS.items():
        try:
            actual = version(pkg)
            if actual != expected:
                mismatches.append(f"{pkg}: expected {expected}, got {actual}")
        except PackageNotFoundError:
            mismatches.append(f"{pkg}: not installed")

    if mismatches:
        print("FATAL: Dependency version mismatch", file=sys.stderr)
        for m in mismatches:
            print(f"  {m}", file=sys.stderr)
        sys.exit(1)

if __name__ == '__main__':
    verify_runtime_versions()
    # ... rest of application

This catches the case where someone manually installed a different version after the Docker build, or where a package got upgraded by a transitive dependency’s post-install script (yes, this can happen).

Why uv and Poetry Don’t Fully Solve This

Tools like uv and Poetry promise better dependency resolution. They do help — uv is dramatically faster, Poetry has a proper SAT solver. But they can’t fix the fundamental gaps:

• Binary compatibility: No resolver can detect that your CUDA version doesn’t match the wheel’s requirements unless the wheel metadata declares it (most don’t).
• Runtime-only dependencies: If a package imports X.Y.Z without declaring X as a dependency in metadata, no resolver will catch it.
• Platform differences: You still need to lock per-platform unless you only use pure-Python wheels.

I’ve migrated several projects to uv (see my earlier post for benchmarks). The speed improvement is real — 10x faster installs. But I still use hash verification and runtime checks.

What I’d Do Differently Next Time

If I were starting a new Python project today:

1. Use uv for speed, pip-tools for locking. uv for day-to-day installs, pip-compile --generate-hashes for production lock files.
2. Lock per platform. requirements-linux-x86_64.txt, requirements-linux-arm64.txt, requirements-mac.txt.
3. Verify imports in CI AND at application startup. Catch failures before deployment, but also catch post-deployment corruption.
4. Pin Python patch version. Not python:3.11, but python:3.11.7-slim-bullseye. Patch releases can change C extension ABI.
5. Use --no-binary for critical security packages. Forces source builds, so you see compilation errors if the environment doesn’t match.

The debugging time saved by catching these issues in CI vs production is worth the extra 30 seconds in build time.

If you’re tired of dependency debugging at 2am, grab some Dark Chocolate Espresso Beans and set up hash verification tonight. Your future self will thank you.

FAQ

Q: Can I use pip freeze instead of pip-compile?

pip freeze only captures what’s currently installed, without hashes or platform info. If you run it on macOS and deploy to Linux, you’ll get different wheels. pip-compile generates a lock file from declared dependencies with full hash verification, making it reproducible across platforms.

Q: Why do some packages install fine locally but fail in Docker?

Usually missing system libraries (e.g., libxml2-dev for lxml) or different glibc versions. Your local OS has them installed from previous unrelated work; the minimal Docker base image doesn’t. The fix: explicitly install build dependencies in your Dockerfile before pip install, or use --no-binary to force source builds that will fail loudly if libs are missing.

Q: How do I debug a segfault during import with no traceback?

Run Python with -X dev to enable debug mode and faulthandler: python -X dev -X faulthandler app.py. This prints the C stack trace. Usually it’s a binary wheel built against a different libc or CUDA version. Check ldd on the .so file to see missing symbols: ldd /path/to/package.so.