Understanding Financial Model Failures

Essay

The Investor Problem

This month built two of finance’s most elegant structures; the CAPM’s clean line relating risk to reward, and the regression machinery that estimates it and, in the same breath, catalogued their failures. The pattern is not unique to one model. It is the signature experience of quantitative finance: frameworks of genuine mathematical beauty, built by brilliant people, validated on decades of data that fail, (sometimes catastrophically,) when reality is asked to cooperate. Portfolio insurance failed in 1987. The Nobel-staffed fund LTCM failed in 1998. The risk models of the world’s largest banks failed together in 2008.

The lazy responses are both wrong: “models are useless” ignores that every serious investor uses them anyway, profitably; “the models just need improving” ignores that the failures repeat with each improved generation. The productive question is structural: why do financial models fail, and why fail in the particular, patterned ways they do? This essay offers the three-part answer the field has painfully assembled: the mathematics assumes tails thinner than reality’s, worlds more stable than reality’s, and people simpler than real ones. Understanding all three is the beginning of using models wisely.

First Failure: The Tails Are Fat

Most classical finance mathematics rests, openly or quietly, on the normal distribution; the bell curve under which extreme events are vanishingly rare. The convenience is enormous: normality makes portfolio math tractable, reduces risk to a single number (variance), and undergirds everything from Markowitz optimization to option pricing to Value-at-Risk.

Reality declined the assumption. Market returns are fat-tailed: extreme moves occur orders of magnitude more often than the bell curve permits. Under normality, a one-day crash of the size seen in October 1987 should not have occurred once in the lifetime of the universe; markets have produced events of loosely that improbability several times in a century. Benoit Mandelbrot documented this in cotton prices as early as the 1960s. The finding was politely admired and largely set aside, because abandoning normality meant abandoning tractability.

The failure mode this creates is insidious: models calibrated on ordinary days are precisely wrong about the days that matter. A risk estimate built on years of calm data does not merely underestimate catastrophe; it underestimates it most confidently just before it arrives, because the calm sample contains no warning. The portfolio “safe at the 99% level” fails not at the 1% rate but far more often, and the failures cluster. Fat tails are why diversification disappoints in crashes, why volatility-based risk measures lull, and why the prudent add margins of safety that no formula requested.

Second Failure: The World Changes Regimes

Every statistical model makes a wager so basic it is rarely stated: that the future will be drawn from the same distribution as the training data, that the world is stationary. Financial history is a record of that wager losing. Inflation regimes flip the stock-bond correlation from negative to positive, quietly breaking every “balanced” portfolio built on the old sign. Currency pegs hold for a decade but then break in an afternoon. Volatility itself switches between long placid regimes and violent ones. The parameters we estimate such as betas, correlations, premia, are not constants of nature; they are summaries of a particular era, and eras end.

Worse, in finance the model itself helps end them. Markets are reflexive: a model that identifies a profitable pattern attracts capital to that pattern, which reshapes the prices that generated it. The strategy crowded becomes the strategy broken, sometimes gently (premia decaying after publication), sometimes violently (a crowd of similar models unwinding together, as in 1998 and 2007, each fund’s “independent” risk system commanding the same sale at the same moment). Physics does not rearrange itself because physicists publish; markets do. This is the deepest This is the deepest disanalogy between financial modelling and natural science, and no amount of data cures it.  between financial modeling and natural science, and no amount of data cures it. More history simply means more regimes averaged into mush.

Third Failure: The Humans Inside the Machine

The third failure operates through people (twice).

The models assume rational actors, and Month 6 of this blog will detail how systematically real investors deviate: panicking at bottoms, herding at tops, overweighting stories against arithmetic. When behaviour is correlated and fear is the most contagious phenomenon in finance, the deviations do not cancel; they become the fat tails and regime shifts of the first two failures.  behavior is correlated and fear is the most contagious phenomenon in finance, the deviations do not cancel; they become the fat tails and regime shifts of the first two failures. The crash is not an input the model missed; it is the aggregated behavior of people the model assumed away.

But the subtler human failure is the modeler’s own. A model is also a psychological object: its precision feels like knowledge, its output launders judgment into apparent objectivity, and its users, under career pressure to act confidently usually treat the number as more real than the assumptions beneath it. Risk models before 2008 did not merely fail technically; they licensed the leverage that made their failure catastrophic, because a formal number saying “safe” outranks a human saying “uneasy” in every institutional Risk models before 2008 did not merely fail technically; they licensed the leverage that made their failure catastrophic, because a formal number saying “safe” outranks a human saying “uneasy” in every institutional argument. Goodhart’s law completes the trap: once a model’s number becomes the target, the VaR limit, the rating, the tracking error, the organisation optimises against the number’s blind spots, stuffing risk precisely where the model cannot see it. The failure is then not against the model but through it.

Limitations and Honest Complications

An essay on model failure rightly acknowledges its shortcomings.  While models fail, unaided intuition fails even worse.  It lacks an audit trail, stated assumptions and a learning mechanism.  This month’s disciplines, writing down claims, estimating them and bounding errors, remain the most honest technology investing has.  Furthermore, the field isn’t unfamiliar with these failures.  Fat-tailed distributions, regime-switching models and behavioural finance are decades old and each addresses real issues. However, they often come at the cost of estimating more parameters from limited historical data, sophistication isn’t synonymous with safety.  Some model “failures” stem from user errors, such as applying instruments outside their design range or extrapolating without understanding the assumptions.  The pencil isn’t responsible for the forged signature.

The honest summary is George Box’s, now a widely recognised proverb: all models are wrong but some are useful and the usefulness depends on the user understanding their limitations.

The Long-Term Lesson

The three failures share a moral: models break at exactly the point where confidence in them peaks, in the tails, at regime boundaries, under crowd stress, because those are the places their assumptions quietly excluded. The thoughtful investor’s response is not to discard the instruments but to hold them the way a good sailor holds a weather forecast: gratefully, skeptically, and with the vessel built for conditions the forecast cannot imagine.

In practice that means three standing habits, which this blog will apply for the rest of the year. Use models to structure thinking, not to replace it. The discipline of explicit assumptions is the true product; the decimal output is a by-product. Size everything for the model being wrong, margins of safety, position limits, and reserves are the premium paid for the insurance no formula sells. And watch the humans, including the one running the model, because every quantitative failure in market history was, at bottom, a human deciding to believe a number more than the number deserved.

Models fail. That is not the scandal. The scandal is forgetting that they must, and building, on top of that forgetting, positions that cannot survive the reminder. The investor who remembers is free to use every tool in the box, because none of them is holding up the house.