Data Mining Finds Patterns That Were Never There

Searching Versus Testing

There are two ways to reach a conclusion. hypothesis test you propose a relationship for a reason and then check to see if the data supports it.en data mining looks for relationships in the data and reports what it finds

The reported statistics seem identical. What differs is how many possibilities were examined to arrive at them and that difference determines whether the result means anything

Note what is not part of this distinction. No one has to be careless biased or dishonest for a returned result to be false. Falsehood is a property of the search rather than the searcher. A completely rigorous analyst applying correct methods to clean data without any agenda will produce impressive nonsense if the search space is large enough and will produce it reliably rather than occasionally

The Classic Absurdities

Literature on spurious correlations exists to illustrate this. Butter production in a small country has been shown to closely track a major stock index over a period of several years. The number of letters in spelling bee winning words has tracked spider deaths

These relationships are real in the data and are completely meaningless. They exist because someone looked at a huge number of series for pairs that moved together and with enough series those pairs are guaranteed

No one is fooled by the production of butter. The problem is that the same process produces relationships that are plausible enough to believe and those are the dangerous ones

How Fast the Search Space Grows

Guaranteed reason is the right word and probably doesn't come from combinatorics and the growth is faster than intuition expects

Comparing each series with others means examining n multiplied by n minus 1 divided by two different pairs. With 100 series that is 4,950 comparisons. With 1,000 series which is a modest collection of data by any current standard it is 499,500. Multiplying the number of entries by ten multiplied the number of candidate relationships by about a hundred

If the usual threshold is applied to those numbers the result is more arithmetic than opinion. If each of the 4,950 pairs is pure noise about 247 of them will pass a 5 percent significance bar. Among the 499,500 pairs about 24,975 will. Nothing needs to be true anywhere in the data set for that to happen

Then you add the dimensions that a real search includes without anyone counting them. Each pair can be tested with several lags since a relation can operate with one lag. Testing twelve lags turns 4,950 comparisons into 59,400 and about 2,970 of them will appear significant in noise alone. Add a transform option a window option and a subperiod option and the multiplication continues

The strongest relationship found in such a search is not a typical result. It is the maximum of a huge number of extractions and the maximum of a large number of noise extractions always attracts attention. This is what makes the presentation so compelling: the graph shown is the best among half a million selected precisely because it looked like something

Two Series Can Correlate Without Touching

Combinatorics explains how many false relationships appear. A second mechanism explains why individual pairs correlate so strongly and works even for small searches

Any two series that trend in the same direction over a period will correlate regardless of whether they have anything to do with each other. Correlation is measuring the shared trend and nothing more. Over the course of a decade in which prices wages output populations and most economic quantities in general rose a huge number of unrelated pairs are rising together by construction

That's why the famous absurdities tend to be two slowly growing quantities and not two volatile quantities. Butter production and a stock index share no mechanism and do share a direction and more than a decade of shared direction is enough to produce a correlation coefficient that would be impressive if it meant anything

The problem is worse than a nuisance in statistics because a regression between two trend series is also of impressive importance. The observations are highly dependent on each other so the test treats the sample as if it contained much more independent evidence than it does. Very high explanatory power and very reliable statistics can be produced from two series that were generated completely independently of each other

The standard defense is to work with swings rather than levels. Comparing monthly swings removes the shared trend and asks whether the two move together once the trend is removed which is a much harder test to pass by accident. Most spurious relationships built on levels do not survive which is reason to be suspicious when a striking correlation only occurs in levels

Why Finance Is Especially Exposed

The limited point of history is underestimated. There may be decades of daily market data but for questions about market cycles the effective number of independent observations is small. A century of data contains a handful of important cycles not thousands

The incentive row belongs in the table for a reason that's easy to overlook. In most fields an exhaustive search that turns up nothing is a career-neutral publishable result. At an investment services firm it's a product-less quarter so the search doesn't stop when testing runs out. It stops when something is found which turns the incentive into an increase in the number of tests performed and the number of tests performed is exactly the amount that determines how insignificant the surviving result is

That combination is the specific trap. The search space grows with the number of variables which is effectively unlimited while the evidence available for judging grows with the number of independent observations which is severely limited and cannot be expanded by collecting more of the same. More data of the type that is easy to obtain makes the problem worse rather than better because it expands the numerator of the search without increasing the denominator of the evidence

Flexible Models Search Without Being Told To

Modern methods make counting more difficult because the search is no longer a list of comparisons that a person decided to make

A model with many parameters and a flexible functional form is exploring a huge space of possible relationships as part of the fit itself. No one enumerated the candidates and no record records how many were considered but the effective number of hypotheses examined is a property of how expressive the model is and for flexible ones it is very large

That's why a method that can represent almost any pattern will find a pattern in almost any data. Introducing random noise into a sufficiently flexible model does not produce an error message. It produces a fitted model with an impressive sample fit and an available explanation of why it works

The defense is the same as always and is often skipped precisely because the machinery feels rigorous. Restrict the model to something that is no more complex than the data can support and judge it only on the basis of data that did not participate in its adjustment

The Adaptation Problem

Finance has a difficulty that most sciences do not. Even a genuine relationship can stop working once it is known because the participants act on it and the action eliminates the opportunity

Documented anomalies have often weakened after publication. This means that out-of-sample failure does not clearly distinguish a false discovery from a real effect that has been ruled out eliminating the most useful diagnosis

The consequence is that the discipline of the field cannot be based solely on replication as is the case elsewhere. In a laboratory science a finding that fails to replicate is dead. In finance the same failure is genuinely ambiguous and both readings have defenders generally interested in the answer. That ambiguity is not a temporary state of knowledge. It follows from the fact that the subject reacts to being studied and this is permanent

When Mining Is Legitimate

Exploratory analysis is a reasonable and necessary activity. Looking at data to generate ideas is how hypotheses are formed

The requirement is that search and confirmation use different data. Generate the hypothesis on a sample and then test it on a sample that was not involved in its generation. Machine learning practice formalizes this with training validation and testing sets and the test set must be used once

The reason it must be used once is that repeated query turns it into training data. Each look leaks information and after enough iterations the pool is no longer independent

The leak does not require anyone to put anything on the obstacle. Consulting it seeing a disappointing result adjusting the approach and consulting it again uses reservation to select between approaches which is training along a slower route. After a dozen such cycles the supposedly intact sample has quietly become part of the search and the person running it can honestly say that no model has ever been installed on it

The Filters Worth Applying

It requires a mechanism. If a relationship cannot be explained in terms of behavior risk structure or constraint it is probably an accident

Prefers simple relationships. A pattern that requires many conditions to appear is to describe a small number of historical cases

Consult by markets and periods. An effect present in a country and a decade is weaker evidence than one that appears widely

And count the tests honestly including the abandoned ones because the search space is what determines how impressive the survival result is

The mechanism requirement carries the greatest weight of the four and it works for a reason worth mentioning. Requiring an economic history does not strengthen statistics. It reduces the search space before the search begins by discarding the overwhelming majority of candidate relationships for reasons that have nothing to do with the data. A hypothesis space reduced in advance to things that might be plausible is one in which a surviving finding actually means something

Where the Mechanism Filter Leaks

The filter has a weakness that appears as soon as it is used casually and it is worth understanding this before relying on it

Economic explanations are abundant and cheap. Show an experienced investor almost any pattern and within a minute a plausible explanation of it will be available based on investor behavior risk compensation market structure regulatory constraints or the incentives of some category of participant. The supply of available mechanisms is effectively unlimited so the requirement to produce one is not much of a constraint when applied after the pattern has been found

The filter only works when the story appears first. A mechanism written before the search is a real constraint because it commits a specific relationship beforehand and may be erroneous. A subsequently produced mechanism has been selected to fit a pattern that is already known to exist so it cannot fail to fit and it provides no evidence

The practical version is to ask when the explanation was written rather than whether it is convincing. Convincing is not the proof since a obtained result will always be accompanied by a convincing explanation. A specific prediction made in advance and then confirmed is a different type of claim and is the only one that the search space does not undermine

The Bottom Line

Data mining always finds something since sufficient searching guarantees a pattern in any data set. Distinguishing genuine relationships requires an economic mechanism simplicity breadth across markets and periods and confirmation of data that did not participate in the discovery. In finance the problem is compounded because true effects decay once known making sampling failure ambiguous rather than decisive. The number that determines whether a finding is worth anything is the one that no one records: how many relationships were available to be found before this one was found.presented