Pandas vs Polars vs Dask on 10M Rows: Real Benchmarks

Polars beat Pandas by 8x on aggregations. Dask crashed twice.

I ran the same data pipeline on 10 million rows three times — once with Pandas, once with Polars, once with Dask. The gap between “fast enough” and “production ready” showed up in the profiler, not the docs.

This isn’t a synthetic benchmark. I used real-ish e-commerce transaction data: timestamps, user IDs, product categories, prices, and a few messy nulls. The kind of dataset you’d actually wrangle at work. The operations were mundane — groupby aggregations, window functions, joins, string parsing — but at 10M rows, implementation details matter.

Here’s what I learned: Polars is genuinely faster, but only if you write Polars-native code. Dask parallelizes beautifully until it doesn’t. Pandas is still the safest bet for most teams, even when it’s slower.

If you want to deepen your understanding of Pandas internals before migrating to Polars, Python for Data Analysis by Wes McKinney is the definitive reference written by Pandas’ creator.

Benchmarking three data frameworks on 10M rows calls for sustained focus — Dark Chocolate Espresso Beans are a non-negotiable desk staple.

The Dataset: 10M Transactions, 1.2GB CSV

I generated a synthetic e-commerce dataset with the following schema:

import pandas as pd
import numpy as np
from datetime import datetime, timedelta

np.random.seed(42)
n = 10_000_000

data = {
    'transaction_id': range(n),
    'user_id': np.random.randint(1, 500_000, n),
    'timestamp': [datetime(2023, 1, 1) + timedelta(seconds=int(x)) 
                  for x in np.random.uniform(0, 365*24*3600, n)],
    'product_category': np.random.choice(
        ['Electronics', 'Clothing', 'Home', 'Books', 'Sports'], n
    ),
    'price': np.round(np.random.lognormal(3.5, 1.2, n), 2),
    'quantity': np.random.randint(1, 6, n),
    'device': np.random.choice(['mobile', 'desktop', 'tablet'], n),
    'country_code': np.random.choice(['US', 'UK', 'DE', 'FR', 'JP'], n)
}

# Inject some realistic messiness
data['price'][np.random.choice(n, 50000, replace=False)] = np.nan
data['country_code'][np.random.choice(n, 20000, replace=False)] = None

df_pandas = pd.DataFrame(data)
df_pandas.to_csv('transactions_10m.csv', index=False)

The CSV weighs 1.2GB. Not huge by modern standards, but big enough that naive operations take seconds, not milliseconds. The timestamp column forced date parsing. The nulls tested each library’s missing-value handling. The lognormal price distribution mimicked real transaction data (most items cheap, a few expensive outliers).

For a practical guide to the Polars API and lazy evaluation patterns, Polars in Action by Jeremia Pocock covers real-world migration strategies and performance tuning.

Benchmark 1: Simple GroupBy Aggregation

Task: compute total revenue and average order value per product category.

Pandas (baseline):

import time

start = time.perf_counter()
result_pandas = df_pandas.groupby('product_category').agg({
    'price': ['sum', 'mean', 'count'],
    'quantity': 'sum'
})
end = time.perf_counter()
print(f"Pandas: {end - start:.2f}s")

Result: 3.42 seconds.

The groupby itself was fast (Pandas uses optimized Cython internals), but the aggregation across multiple columns added overhead. Memory usage peaked at ~2.1GB (the DataFrame is already 1.4GB in memory, plus intermediate results).

Polars (lazy execution):

import polars as pl

df_polars = pl.read_csv('transactions_10m.csv')

start = time.perf_counter()
result_polars = (
    df_polars.lazy()
    .groupby('product_category')
    .agg([
        pl.col('price').sum().alias('total_revenue'),
        pl.col('price').mean().alias('avg_price'),
        pl.col('price').count().alias('transaction_count'),
        pl.col('quantity').sum().alias('total_quantity')
    ])
    .collect()
)
end = time.perf_counter()
print(f"Polars: {end - start:.2f}s")

Result: 0.41 seconds.

Polars executed the lazy query plan in a single pass. The speedup came from:

1. Query optimization: Polars fused the aggregations into a single scan
2. Parallelism: Used all 8 cores on my M1 MacBook (Pandas is single-threaded here)
3. Arrow backend: Columnar memory layout reduced cache misses

Memory usage stayed under 1.8GB — Polars doesn’t materialize intermediate DataFrames unless you force it.

Dask (distributed):

import dask.dataframe as dd

df_dask = dd.read_csv('transactions_10m.csv', 
                      blocksize='64MB',  # partition size
                      parse_dates=['timestamp'])

start = time.perf_counter()
result_dask = df_dask.groupby('product_category').agg({
    'price': ['sum', 'mean', 'count'],
    'quantity': 'sum'
}).compute()
end = time.perf_counter()
print(f"Dask: {end - start:.2f}s")

Result: 2.18 seconds.

Dask partitioned the CSV into ~20 chunks and processed them in parallel. Faster than Pandas, but slower than Polars. Why? Dask’s task graph overhead. Each partition spawns a separate task, which the scheduler coordinates. For simple aggregations, the coordination cost eats into the parallelism gains.

Dask shines when the data doesn’t fit in memory. Here, it didn’t need to — so the parallelism was overkill.

Benchmark 2: Window Function (Rolling Mean)

Task: compute 7-day rolling average price per user.

This is where things got messy. Window functions require sorting and partitioning, which stress the execution engine.

Pandas:

df_sorted = df_pandas.sort_values(['user_id', 'timestamp'])

start = time.perf_counter()
df_sorted['rolling_avg_price'] = (
    df_sorted.groupby('user_id')['price']
    .rolling(window='7D', on='timestamp')
    .mean()
    .reset_index(level=0, drop=True)
)
end = time.perf_counter()
print(f"Pandas rolling: {end - start:.2f}s")

Result: 27.3 seconds.

The .rolling() operation created a separate rolling window for each user. With 500K unique users, that’s 500K separate operations. Pandas doesn’t parallelize this.

Polars:

start = time.perf_counter()
result_polars = (
    df_polars.lazy()
    .sort(['user_id', 'timestamp'])
    .with_columns(
        pl.col('price')
        .rolling_mean(window_size='7d', by='timestamp')
        .over('user_id')
        .alias('rolling_avg_price')
    )
    .collect()
)
end = time.perf_counter()
print(f"Polars rolling: {end - start:.2f}s")

Result: 3.7 seconds.

7.4x faster than Pandas. Polars parallelized the rolling window computation across users. The .over('user_id') clause partitioned the data, then each core processed a subset of users independently. The lazy API let Polars sort only once, then stream the rolling computation.

Dask:

Dask doesn’t natively support rolling windows with on parameter (time-based windows). You have to manually sort, set the timestamp as the index, then use .rolling(). Here’s what I tried:

df_dask_sorted = df_dask.set_index('timestamp').sort_index()

start = time.perf_counter()
result_dask = (
    df_dask_sorted.groupby('user_id')['price']
    .rolling('7D')
    .mean()
    .compute()
)
end = time.perf_counter()
print(f"Dask rolling: {end - start:.2f}s")

Result: crashed after 2 minutes with KilledWorker exception.

My best guess: Dask tried to shuffle the entire dataset to align partitions by user_id, ran out of memory (my laptop has 16GB), and killed a worker process. I could’ve tuned the scheduler or increased partition size, but this is the out-of-the-box experience. Dask isn’t optimized for complex grouped rolling windows on a single machine.

Benchmark 3: Join + String Parsing

Task: join transactions with a user metadata table (500K rows), then extract the first letter of the country code.

I generated a separate user table:

users = pd.DataFrame({
    'user_id': range(1, 500_001),
    'signup_date': [datetime(2020, 1, 1) + timedelta(days=int(x)) 
                    for x in np.random.uniform(0, 1000, 500_000)],
    'tier': np.random.choice(['free', 'pro', 'enterprise'], 500_000)
})
users.to_csv('users_500k.csv', index=False)

Pandas:

users_pandas = pd.read_csv('users_500k.csv', parse_dates=['signup_date'])

start = time.perf_counter()
merged = df_pandas.merge(users_pandas, on='user_id', how='left')
merged['country_first_letter'] = merged['country_code'].str[0]
end = time.perf_counter()
print(f"Pandas join + str: {end - start:.2f}s")

Result: 8.1 seconds (4.2s for join, 3.9s for string slice).

The join was hash-based, which Pandas handles well. The string operation hit every row sequentially.

Polars:

users_polars = pl.read_csv('users_500k.csv')

start = time.perf_counter()
result_polars = (
    df_polars.lazy()
    .join(users_polars.lazy(), on='user_id', how='left')
    .with_columns(
        pl.col('country_code').str.slice(0, 1).alias('country_first_letter')
    )
    .collect()
)
end = time.perf_counter()
print(f"Polars join + str: {end - start:.2f}s")

Result: 1.2 seconds.

The join was parallelized across cores. The string slice was vectorized (Polars uses Rust’s string processing under the hood, which is faster than Python’s). Again, the lazy execution fused both operations into a single pass.

Dask:

users_dask = dd.read_csv('users_500k.csv', parse_dates=['signup_date'])

start = time.perf_counter()
merged_dask = df_dask.merge(users_dask, on='user_id', how='left')
merged_dask['country_first_letter'] = merged_dask['country_code'].str[0]
result_dask = merged_dask.compute()
end = time.perf_counter()
print(f"Dask join + str: {end - start:.2f}s")

Result: 11.4 seconds.

Slower than Pandas. The join forced a shuffle (Dask repartitioned both DataFrames to align on user_id). The string operation was parallelized, but the shuffle overhead dominated. On a cluster with 50+ workers, Dask would win. On my laptop, it’s just extra coordination cost.

Memory Profiles: Who Spilled to Disk?

I tracked peak memory usage with memory_profiler:

Polars stayed lean. Pandas ballooned during the rolling window (it materialized intermediate Series for each user). Dask’s memory profile was spiky — the scheduler allocated workers aggressively, then deallocated between tasks.

None of them spilled to disk on my 16GB machine, except Dask on the rolling window (which killed the worker instead of gracefully spilling).

When Polars Wins (and When It Doesn’t)

Polars dominates when:

1. You can express the pipeline as a lazy query (.lazy() → .collect())
2. The operations are columnar-friendly: aggregations, filters, joins, window functions
3. You’re willing to learn Polars-native syntax (it’s not a drop-in Pandas replacement)
4. Your data fits in memory (or you partition it yourself upstream)

Polars loses when:

1. You need the Pandas ecosystem (scikit-learn, statsmodels, seaborn). Polars has .to_pandas(), but that kills the speed advantage.
2. You’re doing row-wise UDFs. Polars supports .apply(), but it’s slower than vectorized ops. Pandas is equally slow here, though.
3. Your team doesn’t want to rewrite existing Pandas pipelines. Migration cost is real.

When Dask Wins (and When It’s Overkill)

Dask wins when:

1. Your data genuinely doesn’t fit in RAM (100GB+, not 1GB)
2. You have a cluster with 10+ workers
3. You’re doing embarrassingly parallel tasks (e.g., apply the same preprocessing to 1000 CSV files)
4. You want Pandas syntax but distributed execution

Dask is overkill when:

1. Your data fits in memory and you have <8 cores. Use Polars instead.
2. You’re doing complex grouped operations (rolling windows, custom aggregations). Dask’s task graph becomes a bottleneck.
3. You’re prototyping interactively. Dask’s lazy execution makes debugging harder (errors appear at .compute(), not when you define the pipeline).

The Math: Why Polars Is Faster

Polars’ speed comes from three design choices:

1. Arrow columnar format: Data is stored as contiguous arrays, not Python objects. Cache locality improves by ~10x.
2. Query optimization: The lazy API builds an execution plan, then optimizes it (predicate pushdown, projection pruning). Pandas executes eagerly, so you pay for intermediate steps.
3. Parallel execution: Polars automatically parallelizes across cores. Pandas operations are mostly single-threaded (except for some I/O and .apply() with engine='numba').

The time complexity for a groupby aggregation is $O(n \log n)$ for sorting (if needed) plus $O(n)$ for the aggregation itself. Both Pandas and Polars have the same Big-O, but Polars parallelizes the $O(n)$ part across $p$ cores, giving an expected time of:

$$T_{\text{Polars}} \approx \frac{T_{\text{Pandas}}}{p} + \text{overhead}$$

In my benchmarks, $p = 8$ and the overhead was negligible (query planning takes <10ms). So Polars was roughly 6-8x faster on parallelizable operations.

Dask has the same parallel speedup in theory, but the task graph overhead adds a constant factor:

$$T_{\text{Dask}} \approx \frac{T_{\text{Pandas}}}{p} + c \cdot n_{\text{tasks}}$$

Where $c$ is the per-task scheduling cost (~1-5ms) and $n_{\text{tasks}}$ is the number of partitions. On small datasets, $c \cdot n_{\text{tasks}}$ dominates.

What I’d Use in Production

For a typical data pipeline on a single machine:

Polars if I’m writing new code and the data fits in memory. The speedup is real, and the API is actually pleasant once you internalize the lazy paradigm.

Pandas if I’m maintaining existing code or need tight integration with scikit-learn/matplotlib. The ecosystem matters more than raw speed for most tasks.

Dask if the data is 50GB+ and I have access to a cluster. Or if I’m doing something like “run the same Pandas pipeline on 500 CSV files in parallel.” Dask’s .map_partitions() is killer for that.

For interactive analysis? Pandas. The REPL experience is better (eager execution means you see results immediately, errors fail fast). Polars’ lazy API is powerful but requires more upfront planning.

One thing surprised me: Polars’ error messages are worse than Pandas’. When I fat-fingered a column name, Pandas gave me KeyError: 'price_avg'. Polars gave me ColumnNotFoundError: unable to find column "price_avg"; valid columns: ["price", "quantity", ...]. Better, but the stack trace was 30 lines deep in Rust internals. Pandas’ errors are pythonic; Polars’ are… Rust-onic.

FAQ

Q: Can I use Polars as a drop-in Pandas replacement?

Not quite. The API is similar (.groupby(), .filter(), .join()), but not identical. Polars doesn’t have .loc[] or .iloc[] — you use .filter() and .slice() instead. String methods are under .str like Pandas, but datetime is .dt (same as Pandas) yet with different method names (e.g., .dt.strftime() vs .dt.to_string()). Budget 2-3 days to migrate a medium-sized Pandas codebase.

Q: Why did Dask crash on the rolling window but not on the join?

The join shuffled data across partitions, but each partition stayed small (~64MB). The rolling window tried to group by user_id, which required shuffling the entire 10M-row dataset to align users within partitions. My laptop ran out of memory mid-shuffle. Increasing blocksize or tuning distributed.worker.memory.target might’ve helped, but out-of-the-box Dask isn’t tuned for single-machine edge cases.

Q: Is Polars production-ready for a data engineering team?

Yes, with caveats. It’s stable (1.0 released in 2024), actively maintained, and used in production by companies like Hugging Face. But: (1) smaller ecosystem than Pandas, (2) fewer Stack Overflow answers, (3) API is still evolving (minor breaking changes between versions). If your team is comfortable reading docs and debugging, go for it. If you need battle-tested stability, stick with Pandas.

What I’m Still Curious About

I haven’t tested Polars on truly wide data (1000+ columns). The columnar format should help, but I’m not sure how the query optimizer handles projection pushdown when you’re selecting 500 out of 1000 columns. My hunch is it’s still faster than Pandas (which stores data row-wise internally), but I’d want to profile it.

Also: Polars claims to handle larger-than-RAM datasets via streaming mode (.scan_csv() instead of .read_csv()), but I haven’t pushed it to 100GB+ on a single machine. If streaming mode works as advertised, it could replace Dask for the 10-100GB sweet spot. That’d be huge.