Data Mining Finds Patterns That Were Never There
With enough data and enough computing power, you can find a relationship between almost any two things. The difficulty is telling those apart from the real ones.
Searching Versus Testing
There are two ways to arrive at a finding. In hypothesis testing, you propose a relationship for a reason and then check whether the data supports it. In data mining, you search the data for relationships and report what you find.
The statistics reported look identical. What differs is how many possibilities were examined to arrive at them, and that difference determines whether the result means anything.
The same correlation coefficient means something very different depending on whether it was predicted in advance or selected from ten thousand candidates.
The Classic Absurdities
The literature of spurious correlations exists to make this vivid. Butter production in a small country has been shown to track a major stock index closely over a multi year period. Numbers of letters in winning spelling bee words have tracked deaths by spiders.
These relationships are real in the data and completely meaningless. They exist because someone searched an enormous number of series for pairs that happened to move together, and with enough series such pairs are guaranteed.
Nobody is fooled by butter production. The problem is that the same process produces relationships that are plausible enough to believe, and those are the dangerous ones.
Why Finance Is Especially Exposed
| Condition | Effect |
|---|---|
| Enormous available data | Many candidate relationships |
| Low signal to noise | Real effects are small and hard to see |
| Short usable history | Limited independent observations |
| Strong financial incentive | Rewards for finding something |
| Non stationary relationships | Real effects also decay |
The limited history point is underappreciated. There may be decades of daily market data, but for questions about market cycles the effective number of independent observations is small. A century of data contains a handful of major cycles, not thousands.
The Adaptation Problem
Finance has a difficulty that most sciences do not. Even a genuine relationship can stop working once it is known, because participants act on it and the action removes the opportunity.
Documented anomalies have frequently weakened after publication. This means failure out of sample does not cleanly distinguish a false discovery from a real effect that has been arbitraged away, which removes the most useful diagnostic.
When Mining Is Legitimate
Exploratory analysis is a reasonable and necessary activity. Looking at data to generate ideas is how hypotheses get formed.
The requirement is that the search and the confirmation use different data. Generate the hypothesis on one sample, then test it on a sample that played no part in generating it. Machine learning practice formalises this with training, validation, and test sets, and the test set is meant to be used once.
The reason it is meant to be used once is that repeated consultation turns it into training data. Each look leaks information, and after enough iterations the holdout is no longer independent.
The Filters Worth Applying
Require a mechanism. If a relationship cannot be explained in terms of behaviour, risk, structure, or constraint, it is probably an accident.
Prefer simple relationships. A pattern requiring many conditions to appear is describing a small number of historical instances.
Check across markets and periods. An effect present in one country and one decade is weaker evidence than one appearing broadly.
And count the tests honestly, including the abandoned ones, because the search space is what determines how impressive the surviving result is.
The Bottom Line
Data mining always finds something, since sufficient search guarantees a pattern in any dataset. Distinguishing genuine relationships requires an economic mechanism, simplicity, breadth across markets and periods, and confirmation on data that took no part in the discovery. In finance the problem is compounded because real effects decay once known, which makes out of sample failure ambiguous rather than decisive.