The Sharpe Ratio Trap
Most portfolio optimizers fail because they optimize for the wrong thing. Maximizing Sharpe ratio sounds great until you realize you’ve built a portfolio that’s 80% one stock—technically optimal, but completely useless in practice.
The classic mean-variance optimization from Markowitz (1952) gives you mathematically perfect portfolios that are wildly unstable. Change your historical window by a week and your entire allocation flips. This isn’t a bug in your code; it’s a fundamental issue with unconstrained quadratic optimization on noisy financial data.
Here’s what actually works: constrained optimization with sensible bounds, combined with robust covariance estimation. We’ll build a portfolio optimizer that doesn’t just spit out numbers—it produces allocations you’d actually trade.
Expected Returns Are a Lie (But We Need Them Anyway)
The expected return vector is the most uncertain input in portfolio optimization, yet it drives most of the allocation decision. Historical mean returns are notoriously unstable—the standard error on a 5-year daily mean return is often larger than the mean itself.
I’ve found three approaches that work reasonably well in practice. First, use implied returns from market capitalization weights (the Black-Litterman approach). Second, shrink historical means toward a common value—typically the market return. Third, just assume equal returns and optimize purely on risk, which sounds lazy but often outperforms more sophisticated methods.
For this engine, we’ll use a simple exponential weighting scheme that gives more weight to recent data:
import numpy as np
import pandas as pd
from scipy.optimize import minimize
import yfinance as yf
def calculate_exponential_returns(prices, span=180):
"""
Calculate expected returns with exponential weighting.
Recent data gets more weight (span=180 means ~6 months half-life).
"""
returns = prices.pct_change().dropna()
# Exponential weights: more recent = higher weight
weights = np.exp(np.linspace(-1, 0, len(returns)))
weights = weights / weights.sum()
expected_returns = (returns * weights.reshape(-1, 1)).sum(axis=0)
# Annualize (assuming daily data)
return expected_returns * 252
# Load data from our Part 1 pipeline
tickers = ['AAPL', 'MSFT', 'GOOGL', 'AMZN', 'TSLA', 'JPM', 'V', 'WMT', 'PG', 'JNJ']
data = yf.download(tickers, start='2020-01-01', end='2024-12-31', progress=False)
prices = data['Adj Close']
mu = calculate_exponential_returns(prices)
print("Expected annual returns:")
print(mu.sort_values(ascending=False))
On my system (Python 3.11, yfinance 0.2.36), this outputs something like:
Expected annual returns:
TSLA 0.245
AAPL 0.198
GOOGL 0.187
AMZN 0.175
...
These numbers look suspiciously high because we’re coming off a bull market. In practice, you’d probably shrink these toward 8-10% (historical equity premium).
Covariance Matrices and Why They Break
The covariance matrix determines your diversification benefit. With assets, you need to estimate parameters from historical data. For 10 stocks, that’s 55 numbers. For 100 stocks, it’s 5,050.
Sample covariance matrices are terrible for optimization. They’re noisy, often singular or nearly singular, and they overfit to your historical window. The optimizer exploits this noise to create extreme positions—huge longs and shorts that look diversified on paper but are actually betting on small estimation errors.
Ledoit-Wolf shrinkage (Ledoit and Wolf, 2004) fixes this by shrinking the sample covariance toward a structured target, typically a constant correlation matrix. The shrinkage intensity is chosen to minimize expected estimation error. It’s not perfect, but it’s dramatically better than raw sample covariance.
from sklearn.covariance import LedoitWolf
def calculate_robust_covariance(prices, span=180):
"""
Ledoit-Wolf shrinkage on exponentially weighted returns.
This gives you a covariance matrix that won't explode during optimization.
"""
returns = prices.pct_change().dropna()
# Exponential weighting on returns
weights = np.exp(np.linspace(-1, 0, len(returns)))
weights = weights / weights.sum()
weighted_returns = returns.values * np.sqrt(weights).reshape(-1, 1)
# Ledoit-Wolf shrinkage
lw = LedoitWolf()
lw.fit(weighted_returns)
# Annualize (daily -> annual)
cov_matrix = lw.covariance_ * 252
return pd.DataFrame(cov_matrix, index=prices.columns, columns=prices.columns)
Sigma = calculate_robust_covariance(prices)
# Quick sanity check: are volatilities reasonable?
vols = np.sqrt(np.diag(Sigma))
print("\nAnnualized volatilities:")
print(pd.Series(vols, index=prices.columns).sort_values(ascending=False))
TSLA usually shows up around 40-50% annualized vol, while something like PG is closer to 15-20%. If you see anything over 100% or under 5%, something’s wrong with your data.
The Optimization Problem (With Actual Constraints)
The textbook mean-variance optimization is:
subject to , where is the weight vector and is risk aversion. But this is useless without additional constraints.
Add these:
- Box constraints: (long-only, max 25% per position)
- Minimum position: if then (avoid dust positions)
- Target return: (if you want a specific return level)
The minimum position constraint is annoying to implement as a mixed-integer program, so in practice I just round small weights to zero afterward.
def optimize_portfolio(mu, Sigma, target_return=None, max_weight=0.25):
"""
Mean-variance optimization with sensible constraints.
Returns portfolio weights, expected return, and volatility.
"""
n_assets = len(mu)
# Objective: maximize return - risk (equivalently, minimize -return + risk)
# We use lambda=1 for equal weighting of return and risk
def objective(w):
portfolio_return = w @ mu
portfolio_variance = w @ Sigma @ w
# Return negative because we're minimizing
return -portfolio_return + portfolio_variance
# Constraints
constraints = [
{'type': 'eq', 'fun': lambda w: np.sum(w) - 1} # weights sum to 1
]
if target_return is not None:
constraints.append({
'type': 'ineq',
'fun': lambda w: w @ mu - target_return # return >= target
})
# Bounds: long-only, max weight per asset
bounds = tuple((0, max_weight) for _ in range(n_assets))
# Initial guess: equal weight
w0 = np.ones(n_assets) / n_assets
result = minimize(
objective,
w0,
method='SLSQP', # Sequential Least Squares Programming
bounds=bounds,
constraints=constraints,
options={'maxiter': 1000, 'ftol': 1e-9}
)
if not result.success:
print(f"Warning: Optimization failed - {result.message}")
weights = result.x
# Round small positions to zero (avoid dust)
weights[weights < 0.02] = 0
weights = weights / weights.sum() # renormalize
portfolio_return = weights @ mu
portfolio_vol = np.sqrt(weights @ Sigma @ weights)
portfolio_sharpe = portfolio_return / portfolio_vol
return {
'weights': pd.Series(weights, index=mu.index),
'return': portfolio_return,
'volatility': portfolio_vol,
'sharpe': portfolio_sharpe
}
# Optimize with no target return (risk-return tradeoff)
portfolio = optimize_portfolio(mu, Sigma, max_weight=0.25)
print("\n=== Optimal Portfolio ===")
print("\nWeights:")
print(portfolio['weights'][portfolio['weights'] > 0].sort_values(ascending=False))
print(f"\nExpected Return: {portfolio['return']:.1%}")
print(f"Expected Volatility: {portfolio['volatility']:.1%}")
print(f"Sharpe Ratio: {portfolio['sharpe']:.2f}")
Typical output:
Weights:
AAPL 0.250
MSFT 0.250
GOOGL 0.187
JPM 0.152
PG 0.092
JNJ 0.069
Expected Return: 16.3%
Expected Volatility: 18.7%
Sharpe Ratio: 0.87
Notice how it hits the 25% cap on AAPL and MSFT—that’s the constraint working. Without it, you’d see 40-50% allocations.
The Efficient Frontier (Because Everyone Asks For It)
The efficient frontier plots risk vs. return for optimal portfolios. Every point on the frontier is the minimum-volatility portfolio for a given return target. It’s mostly a visualization tool—you don’t actually pick a portfolio by pointing at the curve—but it’s useful for understanding the risk-return tradeoff.
import matplotlib.pyplot as plt
def calculate_efficient_frontier(mu, Sigma, num_points=50):
"""
Generate efficient frontier by optimizing for different target returns.
Returns arrays of returns, vols, and weights.
"""
min_return = mu.min()
max_return = mu.max()
target_returns = np.linspace(min_return, max_return * 0.9, num_points)
frontier_returns = []
frontier_vols = []
frontier_weights = []
for target in target_returns:
try:
port = optimize_portfolio(mu, Sigma, target_return=target, max_weight=0.25)
frontier_returns.append(port['return'])
frontier_vols.append(port['volatility'])
frontier_weights.append(port['weights'])
except:
# Infeasible target (too high given constraints)
continue
return np.array(frontier_returns), np.array(frontier_vols), frontier_weights
frontier_ret, frontier_vol, frontier_w = calculate_efficient_frontier(mu, Sigma)
# Plot
plt.figure(figsize=(10, 6))
plt.scatter(frontier_vol, frontier_ret, c=frontier_ret/frontier_vol,
cmap='viridis', s=30, label='Efficient Frontier')
plt.colorbar(label='Sharpe Ratio')
# Add individual assets
for ticker in mu.index:
asset_vol = np.sqrt(Sigma.loc[ticker, ticker])
asset_ret = mu[ticker]
plt.scatter(asset_vol, asset_ret, marker='x', s=100, color='red', alpha=0.6)
plt.annotate(ticker, (asset_vol, asset_ret), fontsize=8,
xytext=(5, 5), textcoords='offset points')
plt.xlabel('Volatility (Annual)')
plt.ylabel('Expected Return (Annual)')
plt.title('Efficient Frontier with Individual Assets')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('efficient_frontier.png', dpi=150)
plt.show()
The frontier curves upward—higher return requires higher risk. The maximum Sharpe ratio portfolio is where the line from the origin is tangent to the curve. In practice, I’d pick something slightly below max Sharpe to stay away from the constraint boundaries.
Risk Parity and Why It Sometimes Beats Mean-Variance
Risk parity takes a completely different approach: allocate capital so that each asset contributes equally to portfolio risk. It ignores expected returns entirely, focusing only on volatility and correlation.
The risk contribution of asset is:
where is portfolio volatility. Risk parity sets for all pairs.
def calculate_risk_contributions(weights, Sigma):
"""
Calculate each asset's contribution to portfolio risk.
"""
portfolio_vol = np.sqrt(weights @ Sigma @ weights)
marginal_contrib = Sigma @ weights
risk_contrib = weights * marginal_contrib / portfolio_vol
return risk_contrib
def optimize_risk_parity(Sigma):
"""
Risk parity optimization: equal risk contribution from each asset.
"""
n_assets = len(Sigma)
def objective(w):
# Minimize variance in risk contributions
rc = calculate_risk_contributions(w, Sigma.values)
return np.var(rc) # want all RC equal
constraints = [{'type': 'eq', 'fun': lambda w: np.sum(w) - 1}]
bounds = tuple((0.01, 0.50) for _ in range(n_assets))
w0 = np.ones(n_assets) / n_assets
result = minimize(objective, w0, method='SLSQP',
bounds=bounds, constraints=constraints)
weights = result.x
weights = weights / weights.sum()
return pd.Series(weights, index=Sigma.index)
rp_weights = optimize_risk_parity(Sigma)
rp_return = rp_weights @ mu
rp_vol = np.sqrt(rp_weights @ Sigma @ rp_weights)
rp_sharpe = rp_return / rp_vol
print("\n=== Risk Parity Portfolio ===")
print("\nWeights:")
print(rp_weights[rp_weights > 0.02].sort_values(ascending=False))
print(f"\nExpected Return: {rp_return:.1%}")
print(f"Expected Volatility: {rp_vol:.1%}")
print(f"Sharpe Ratio: {rp_sharpe:.2f}")
# Compare risk contributions
rp_rc = calculate_risk_contributions(rp_weights.values, Sigma.values)
print("\nRisk contributions (should be roughly equal):")
print(pd.Series(rp_rc, index=Sigma.index).sort_values(ascending=False))
Risk parity often outperforms mean-variance in practice, especially when expected returns are uncertain. It tends to overweight low-volatility assets (like bonds or defensive stocks) and underweight high-vol growth stocks. The result is a more stable allocation that doesn’t flip around as much when you reoptimize.
Backtesting Your Optimizer (The Part Everyone Skips)
Optimizing on the full dataset tells you nothing. You need an out-of-sample backtest with realistic rebalancing frequency and transaction costs.
Here’s a simple rolling-window backtest:
def backtest_strategy(prices, optimize_func, rebalance_months=3,
lookback_days=252, transaction_cost=0.001):
"""
Backtest a portfolio optimization strategy.
optimize_func: function(mu, Sigma) -> weights
rebalance_months: rebalance every N months
lookback_days: days of history for estimation
transaction_cost: 10 bps per trade (0.1%)
"""
returns = prices.pct_change().dropna()
# Rebalance dates (first day of every N months)
rebalance_dates = returns.resample(f'{rebalance_months}MS').first().index
rebalance_dates = [d for d in rebalance_dates if d in returns.index]
portfolio_value = [1.0] # start with $1
portfolio_weights = None
for i, date in enumerate(returns.index):
if date in rebalance_dates and i >= lookback_days:
# Use lookback window to estimate mu and Sigma
hist_prices = prices.iloc[i-lookback_days:i]
mu = calculate_exponential_returns(hist_prices)
Sigma = calculate_robust_covariance(hist_prices)
# Optimize
new_weights = optimize_func(mu, Sigma)
# Transaction costs
if portfolio_weights is not None:
turnover = np.abs(new_weights - portfolio_weights).sum()
cost = turnover * transaction_cost
portfolio_value[-1] *= (1 - cost)
portfolio_weights = new_weights
if portfolio_weights is not None:
# Daily return
daily_ret = (returns.loc[date] * portfolio_weights).sum()
portfolio_value.append(portfolio_value[-1] * (1 + daily_ret))
else:
portfolio_value.append(portfolio_value[-1])
return pd.Series(portfolio_value, index=returns.index[:len(portfolio_value)])
# Backtest mean-variance
mv_pf = backtest_strategy(
prices,
lambda mu, Sigma: optimize_portfolio(mu, Sigma)['weights'],
rebalance_months=3
)
# Backtest risk parity
rp_pf = backtest_strategy(
prices,
lambda mu, Sigma: optimize_risk_parity(Sigma),
rebalance_months=3
)
# Benchmark: equal weight, monthly rebalance
eq_pf = backtest_strategy(
prices,
lambda mu, Sigma: pd.Series(1/len(mu), index=mu.index),
rebalance_months=1
)
# Plot performance
plt.figure(figsize=(12, 6))
plt.plot(mv_pf.index, mv_pf.values, label='Mean-Variance', linewidth=2)
plt.plot(rp_pf.index, rp_pf.values, label='Risk Parity', linewidth=2)
plt.plot(eq_pf.index, eq_pf.values, label='Equal Weight', linewidth=2, linestyle='--')
plt.xlabel('Date')
plt.ylabel('Portfolio Value ($1 initial)')
plt.title('Backtest: Portfolio Optimization Strategies (2020-2024)')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('backtest_results.png', dpi=150)
plt.show()
# Performance stats
def calc_performance_stats(pf_values):
rets = pf_values.pct_change().dropna()
total_return = (pf_values.iloc[-1] / pf_values.iloc[0]) - 1
annual_return = (1 + total_return) ** (252 / len(rets)) - 1
annual_vol = rets.std() * np.sqrt(252)
sharpe = annual_return / annual_vol
max_dd = (pf_values / pf_values.cummax() - 1).min()
return {
'Total Return': f"{total_return:.1%}",
'Annual Return': f"{annual_return:.1%}",
'Annual Vol': f"{annual_vol:.1%}",
'Sharpe': f"{sharpe:.2f}",
'Max Drawdown': f"{max_dd:.1%}"
}
print("\n=== Backtest Performance ===")
print("\nMean-Variance:")
for k, v in calc_performance_stats(mv_pf).items():
print(f" {k}: {v}")
print("\nRisk Parity:")
for k, v in calc_performance_stats(rp_pf).items():
print(f" {k}: {v}")
print("\nEqual Weight:")
for k, v in calc_performance_stats(eq_pf).items():
print(f" {k}: {v}")
On the 2020-2024 window, mean-variance typically beats risk parity in a bull market (higher beta exposure), but risk parity shows lower drawdowns. Equal weight is surprisingly competitive—it’s hard to beat consistently.
The real test is 2022 (the correction). Check which strategy held up better. My guess is risk parity, but I haven’t run this exact backtest on that period.
What About Transaction Costs and Taxes?
The optimizer doesn’t care about your cost basis or tax situation, which makes it impractical for taxable accounts. You can add turnover penalties to the objective function:
where is the turnover penalty (say, 0.001 for 10 bps cost). This biases the optimizer toward staying close to your current allocation.
For taxes, you’d need to track cost basis per lot and add a penalty for realizing gains. That gets complicated fast. In practice, most people just run the optimizer in tax-deferred accounts and manually manage taxable accounts.
Putting It All Together: A Production-Ready Class
Here’s the full engine packaged as a reusable class:
class PortfolioOptimizer:
def __init__(self, tickers, lookback_days=252, rebalance_months=3):
self.tickers = tickers
self.lookback_days = lookback_days
self.rebalance_months = rebalance_months
self.prices = None
self.mu = None
self.Sigma = None
def fetch_data(self, start_date, end_date):
"""Download price data."""
data = yf.download(self.tickers, start=start_date, end=end_date, progress=False)
self.prices = data['Adj Close']
return self
def estimate_parameters(self, span=180):
"""Estimate expected returns and covariance."""
self.mu = calculate_exponential_returns(self.prices, span=span)
self.Sigma = calculate_robust_covariance(self.prices, span=span)
return self
def optimize(self, method='mean-variance', **kwargs):
"""Optimize portfolio. Methods: 'mean-variance', 'risk-parity', 'equal-weight'."""
if method == 'mean-variance':
result = optimize_portfolio(self.mu, self.Sigma, **kwargs)
return result['weights']
elif method == 'risk-parity':
return optimize_risk_parity(self.Sigma)
elif method == 'equal-weight':
return pd.Series(1/len(self.mu), index=self.mu.index)
else:
raise ValueError(f"Unknown method: {method}")
def backtest(self, method='mean-variance', **kwargs):
"""Backtest strategy."""
optimize_func = lambda mu, Sigma: self.optimize(method=method, **kwargs)
return backtest_strategy(self.prices, optimize_func,
self.rebalance_months, self.lookback_days)
def plot_efficient_frontier(self):
"""Plot efficient frontier."""
frontier_ret, frontier_vol, _ = calculate_efficient_frontier(self.mu, self.Sigma)
plt.figure(figsize=(10, 6))
plt.scatter(frontier_vol, frontier_ret, c=frontier_ret/frontier_vol,
cmap='viridis', s=30)
plt.colorbar(label='Sharpe Ratio')
for ticker in self.mu.index:
vol = np.sqrt(self.Sigma.loc[ticker, ticker])
ret = self.mu[ticker]
plt.scatter(vol, ret, marker='x', s=100, color='red', alpha=0.6)
plt.annotate(ticker, (vol, ret), fontsize=8, xytext=(5, 5),
textcoords='offset points')
plt.xlabel('Volatility')
plt.ylabel('Expected Return')
plt.title('Efficient Frontier')
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
# Usage
opt = PortfolioOptimizer(tickers, lookback_days=252, rebalance_months=3)
opt.fetch_data('2020-01-01', '2024-12-31')
opt.estimate_parameters(span=180)
# Get optimal weights
weights_mv = opt.optimize('mean-variance', max_weight=0.25)
weights_rp = opt.optimize('risk-parity')
print("\nMean-Variance Weights:")
print(weights_mv[weights_mv > 0.02].sort_values(ascending=False))
print("\nRisk Parity Weights:")
print(weights_rp[weights_rp > 0.02].sort_values(ascending=False))
# Backtest
mv_performance = opt.backtest('mean-variance', max_weight=0.25)
print(f"\nMean-Variance Final Value: ${mv_performance.iloc[-1]:.2f}")
# Plot frontier
opt.plot_efficient_frontier()
This gives you a clean API for portfolio optimization, backtesting, and visualization. You can plug it into a Flask app or a Jupyter notebook.
Wrapping the Math in Reality
Portfolio optimization works when you constrain it aggressively and use robust estimation. Unconstrained mean-variance is a mathematical curiosity, not a trading strategy.
The real value isn’t in finding the “optimal” portfolio—it’s in understanding the risk-return tradeoff and avoiding catastrophic concentration. A well-constrained optimizer keeps you diversified and prevents you from loading up on whatever’s been hot lately.
In Part 5, we’ll deploy this as a real-time dashboard with FastAPI and WebSockets. You’ll see live portfolio updates as prices stream in, with automatic rebalancing alerts when your allocation drifts too far from target. That’s where this gets practical—moving from batch optimization to continuous monitoring.
Did you find this helpful?
☕ Buy me a coffee
Leave a Reply