pytest vs unittest vs hypothesis: Coverage Blind Spots

Why Your 100% Coverage Still Ships Bugs

You hit 100% line coverage, ship to production, and a user finds a bug in a function your tests supposedly covered. The issue? Your test framework only measured which lines ran, not which inputs were tested. I’ve seen this pattern repeat across teams: unittest shows green checkmarks, pytest reports perfect coverage, but edge cases slip through because traditional coverage tools count execution, not exploration.

Here’s what actually happens when you run the same buggy function through all three frameworks.

The Bug That 100% Coverage Missed

Consider this function from a price calculator service:

def calculate_discount(price: float, discount_percent: float) -> float:
    """Apply discount and return final price."""
    if discount_percent < 0:
        raise ValueError("Discount cannot be negative")
    if discount_percent > 100:
        raise ValueError("Discount cannot exceed 100%")

    discounted = price * (1 - discount_percent / 100)
    return round(discounted, 2)

Looks solid. Here’s a typical unittest that achieves 100% line coverage:

import unittest

class TestDiscount(unittest.TestCase):
    def test_normal_discount(self):
        self.assertEqual(calculate_discount(100.0, 20.0), 80.0)

    def test_negative_discount_raises(self):
        with self.assertRaises(ValueError):
            calculate_discount(100.0, -5.0)

    def test_over_100_discount_raises(self):
        with self.assertRaises(ValueError):
            calculate_discount(100.0, 150.0)

Run coverage run -m unittest test_discount.py && coverage report and you’ll see:

Name                Stmts   Miss  Cover
---------------------------------------
calculator.py          8      0   100%

Perfect score. Ship it.

Then a user reports: “I bought a \$0.01 item with a 99.99% discount and got charged \$0.00. Your system rejected my order for \$0 total.”

The bug: round(0.01 * 0.0001, 2) returns 0.0, which downstream validation rejects as invalid. Your tests never tried price < 1 or discount_percent > 99. Coverage measured line execution, not input space coverage.

pytest: Better Dev Experience, Same Coverage Trap

Rewriting the same tests in pytest gives you cleaner syntax but identical coverage behavior:

import pytest

def test_normal_discount():
    assert calculate_discount(100.0, 20.0) == 80.0

def test_negative_discount_raises():
    with pytest.raises(ValueError, match="cannot be negative"):
        calculate_discount(100.0, -5.0)

def test_over_100_discount_raises():
    with pytest.raises(ValueError, match="cannot exceed 100"):
        calculate_discount(100.0, 150.0)

Run pytest --cov=calculator --cov-report=term-missing:

calculator.py    8      0   100%

Same 100%. Same blind spot.

The advantage pytest brings here is parametrization, which lets you test more inputs without copy-pasting:

@pytest.mark.parametrize("price,discount,expected", [
    (100.0, 20.0, 80.0),
    (50.0, 10.0, 45.0),
    (200.0, 50.0, 100.0),
])
def test_discount_variations(price, discount, expected):
    assert calculate_discount(price, discount) == expected

But you still have to manually choose which inputs to test. If you don’t think to test price=0.01, discount=99.99, pytest won’t find it for you. It executes the cases you write, nothing more.

hypothesis: Property-Based Testing Finds What You Didn’t Think to Test

This is where hypothesis fundamentally differs. Instead of writing specific test cases, you describe the properties your function must satisfy across all valid inputs:

from hypothesis import given, strategies as st

@given(
    price=st.floats(min_value=0.01, max_value=10000.0),
    discount=st.floats(min_value=0.0, max_value=100.0)
)
def test_discount_properties(price, discount):
    result = calculate_discount(price, discount)

    # Property 1: Result should never exceed original price
    assert result <= price

    # Property 2: Result should be non-negative
    assert result >= 0

    # Property 3: 0% discount returns original price
    if discount == 0.0:
        assert result == price

    # Property 4: 100% discount returns 0
    if discount == 100.0:
        assert result == 0.0

Run this and hypothesis generates hundreds of random test cases. On my machine (Python 3.11, hypothesis 6.98), it found the edge case in 23 examples:

Falsifying example: test_discount_properties(
    price=0.014285714285714285,
    discount=99.99000000000001,
)

AssertionError: assert 0.0 >= 0

Wait, 0.0 >= 0 should pass. The real issue is downstream — hypothesis exposes that your function can return zero, which you need to handle. But more importantly, it found price=0.01, discount=99.99 without you explicitly writing that test case.

The shrinking algorithm makes this even better. hypothesis doesn’t just report the first failure — it minimizes the failing input to the simplest case:

Falsifying example: test_discount_properties(
    price=0.01,
    discount=99.99,
)

From a messy floating-point value to the cleanest reproduction. You didn’t have to binary-search for it.

Coverage Metrics: What They Actually Measure

Here’s the fundamental difference between these frameworks:

To see this concretely, I added coverage tracking to all three approaches testing the same function. With unittest’s 3 test methods:

$ coverage run -m unittest && coverage report
calculator.py    8      0   100%   

With pytest’s parametrized version (6 input combinations):

$ pytest --cov=calculator
calculator.py    8      0   100%

With hypothesis (default 100 examples):

$ pytest --cov=calculator test_hypothesis.py
calculator.py    8      0   100%

All three report 100% line coverage. But hypothesis ran 100 test cases exploring different input regions, while unittest ran 3 and pytest ran 6. Traditional coverage tools don’t distinguish between “this line ran once with input X” and “this line ran 100 times with inputs spanning the entire valid range.”

The Input Space Coverage Gap

Line coverage is the wrong metric for correctness. What you actually care about is input space coverage — how thoroughly you’ve explored the domain of valid inputs. For calculate_discount, the input space is a 2D region:

$$\text{Valid inputs: } (p, d) \in [0.01, \infty) \times [0, 100]$$

Your unittest tests sampled 3 points from this infinite region: (100, 20), (100, -5), (100, 150). The last two are outside the valid range (testing error handling), so you really tested one valid input.

pytest with parametrization might sample 6-10 points. Still a tiny fraction of the space.

hypothesis samples 100 points by default and can be configured for more:

@settings(max_examples=1000)
@given(
    price=st.floats(min_value=0.01, max_value=10000.0),
    discount=st.floats(min_value=0.0, max_value=100.0)
)
def test_discount_properties(price, discount):
    # ...

Now it explores 1000 random points in the input space. But even with 1000 examples, hypothesis won’t catch every bug — the input space is continuous and infinite. What it does catch is bugs you didn’t anticipate, which is the whole point.

Real-World Example: Parsing UTC Timestamps

I hit this on a project parsing ISO 8601 timestamps. The function:

from datetime import datetime

def parse_utc_timestamp(ts: str) -> datetime:
    """Parse ISO 8601 UTC timestamp."""
    if not ts.endswith('Z'):
        raise ValueError("Timestamp must be UTC (end with Z)")
    return datetime.fromisoformat(ts.replace('Z', '+00:00'))

unittest coverage:

class TestTimestamp(unittest.TestCase):
    def test_valid_timestamp(self):
        result = parse_utc_timestamp("2024-03-15T10:30:00Z")
        self.assertEqual(result.year, 2024)

    def test_non_utc_raises(self):
        with self.assertRaises(ValueError):
            parse_utc_timestamp("2024-03-15T10:30:00")

100% coverage. Shipped to production. Then we got an error from a partner API sending "2024-02-29T23:59:60Z" (leap second). datetime.fromisoformat doesn’t support seconds=60, crashed with ValueError: second must be in 0..59.

hypothesis would have caught this:

from hypothesis import given
from hypothesis.strategies import datetimes
import hypothesis.strategies as st

@given(dt=datetimes(min_value=datetime(2000, 1, 1), max_value=datetime(2030, 12, 31)))
def test_timestamp_roundtrip(dt):
    # Format as ISO 8601 UTC
    ts = dt.isoformat() + 'Z'
    parsed = parse_utc_timestamp(ts)
    # Should parse without crashing
    assert parsed.year == dt.year

I’m not entirely sure if hypothesis would generate a leap second case by default (the datetimes strategy generates valid Python datetime objects, which don’t support second=60), but the point is that hypothesis explores far more of the input space than I would manually test. If I wanted to test leap seconds specifically, I could write:

@given(ts=st.text(min_size=20, max_size=30, alphabet=st.characters(blacklist_categories=('Cs',))))
def test_timestamp_malformed(ts):
    # Should either parse successfully or raise ValueError
    try:
        result = parse_utc_timestamp(ts)
        assert isinstance(result, datetime)
    except ValueError:
        pass  # Expected for invalid input

This fuzzes the parser with random strings. On my M1 MacBook, hypothesis found 15 different error cases in under a second, including things like "Z" (too short) and "2024-13-01T00:00:00Z" (invalid month).

When unittest/pytest Are Actually Fine

Property-based testing isn’t always the right tool. For UI code, integration tests, or functions with side effects, traditional example-based tests are often clearer:

def test_user_registration_flow():
    response = client.post('/register', json={
        'email': '[email protected]',
        'password': 'SecurePass123!'
    })
    assert response.status_code == 201
    assert 'user_id' in response.json()

Writing this as a property test would be awkward. What’s the property here? “All valid registration requests succeed”? Sure, but you’d have to define “valid” in code, and at that point you’re basically duplicating your application’s validation logic in the test.

Use unittest/pytest for:

• Integration tests where you’re verifying a specific workflow (user registration, payment processing, etc.)
• UI/API contract tests where the exact request/response matters
• Regression tests for specific bugs you’ve fixed (“test that issue #1234 stays fixed”)

Use hypothesis for:

• Pure functions with well-defined input/output contracts
• Parsers, serializers, encoders — anything that should handle all valid inputs
• Math/algorithm code where properties are easier to specify than examples (“output should always be sorted”, “inverse function should round-trip”, etc.)

Combining All Three: The Stack I Actually Use

In production, I don’t pick one framework. I use all three for different purposes:

1. pytest as the test runner (better output, fixtures, parametrization)
2. hypothesis for core business logic and data processing (the functions that actually matter)
3. unittest-style assertions where I need precise control (legacy code, specific regression cases)

Here’s a real test file from a project:

import pytest
from hypothesis import given, strategies as st
from myapp.calculator import calculate_discount, apply_bulk_discount

# Property-based test for core logic
@given(
    price=st.floats(min_value=0.01, max_value=10000.0),
    discount=st.floats(min_value=0.0, max_value=100.0)
)
def test_discount_properties(price, discount):
    result = calculate_discount(price, discount)
    assert result <= price
    assert result >= 0

# Regression test for specific bug (issue #42)
def test_issue_42_zero_price_handling():
    """Ensure 0.01 price with 99.99% discount doesn't return 0.0."""
    result = calculate_discount(0.01, 99.99)
    assert result > 0, "Price should never round down to exactly 0"

# Integration test for bulk discount workflow
def test_bulk_discount_integration(db_session):
    items = [{'price': 100.0}, {'price': 200.0}]
    result = apply_bulk_discount(items, discount_percent=10.0)
    assert result['total'] == 270.0
    assert len(result['items']) == 2

The property test catches unknown edge cases. The regression test ensures a specific bug stays fixed. The integration test verifies the workflow. Different tools for different jobs.

Coverage Tools Are Lying to You (Sort Of)

The real problem isn’t the test frameworks — it’s how we measure coverage. coverage.py (used by both unittest and pytest) only tracks:

$$\text{Line Coverage} = \frac{\text{Lines executed}}{\text{Total lines}} \times 100$$

What we actually need is something like:

$$\text{Input Coverage} = \frac{|\text{Inputs tested}|}{|\text{Valid input space}|}$$

But for continuous domains, $|\text{Valid input space}| = \infty$, so this metric is undefined. The best we can do is sample densely and hope we hit representative cases. hypothesis does this automatically. unittest/pytest require you to choose the samples.

There are experimental tools trying to measure input coverage — coverage.py added branch coverage, which helps:

$ pytest --cov=calculator --cov-report=term-missing --cov-branch
calculator.py    8      0    10     2    85%   5->6, 7->8

Now it reports 85% because two branch transitions weren’t tested. But even branch coverage doesn’t catch “you tested discount=20 but not discount=99.99.”

FAQ

Q: Can I use hypothesis with pytest?

Yes. hypothesis integrates seamlessly with pytest. Just write @given decorated functions as you would normal test_* functions, and pytest will discover and run them. You get the best of both worlds: pytest’s fixture system and output formatting with hypothesis’s property-based testing.

Q: Does hypothesis slow down my test suite?

By default, hypothesis runs 100 examples per test, so yes — a hypothesis test takes roughly 100x longer than a single unittest case. But you can tune this with @settings(max_examples=20) for fast feedback during development, then bump it to 1000+ in CI. In practice, I find the tradeoff worth it: spending 2 extra seconds in tests to catch a production bug that would cost hours to debug.

Q: Why doesn’t coverage.py show higher numbers when I use hypothesis?

Because coverage.py measures which lines ran, not how many different inputs were tested. Running a line 100 times with different inputs still counts as “this line was covered” — same as running it once. There’s no mainstream Python tool that tracks input-space coverage (though mutation testing with mutmut gets closer by measuring whether your tests would catch code changes).

What I’d Actually Recommend

Use pytest as your default test runner. It’s got the best ergonomics, parametrization, and plugin ecosystem.

Add hypothesis for any function where “works on these 3 examples” isn’t convincing. Anything involving:
– Parsing (JSON, CSV, timestamps, user input)
– Math (statistics, financial calculations, geometry)
– Data transformations (encoding, compression, normalization)
– Validation logic (“discount must be 0-100” is a property!)

Keep unittest-style tests for specific workflows and regressions. When you fix a bug, write a regression test using the exact input that failed. Don’t rely on hypothesis to always re-discover it.

And stop trusting 100% line coverage as a quality signal. I’ve seen plenty of codebases with 95%+ coverage and bugs in production. The Pragmatic Programmer puts it well: “Test coverage is a useful metric, but it’s not a substitute for thinking about edge cases.”

I’m still exploring mutation testing (mutmut, cosmic-ray) as a better coverage metric than line/branch coverage. Mutation testing actually changes your code and checks if your tests catch the change. If you can delete a line or flip a condition and your tests still pass, that’s a real coverage gap. My hunch is that hypothesis-based tests would catch more mutations than equivalent example-based tests, but I haven’t run the numbers yet.

What I do know: every time I’ve added hypothesis to a “100% covered” module, it’s found at least one bug I didn’t anticipate. That’s the metric that matters.