Regression Fits a Line and Tells You Almost Nothing About Why

What It Does

Linear regression finds the straight line that best fits a set of paired observations where it is best to minimize the sum of the squared vertical distances from the points to the line

The result is a slope an intercept and a set of statistics that describe how well the line fits

In finance the most familiar application is beta- Regresses a stock's returns on market returns and the slope is beta. A slope of 1.3 means the stock has historically moved 1.3 percent for every 1 percent market move

That's all arithmetic offers. A slope an intercept and some uncertainty measures around it. Everything else said about a regression is an interpretation superimposed afterwards and arithmetic has no say in whether that interpretation is reasonable. It will fit a line to any two columns of numbers given to it including two completely unrelated columns and report the result in exactly the same reliable format either way

Reading the Output

R-squared is the most commonly interpreted error. An R-squared of 0.4 means that 40 percent of the variation in the dependent variable is associated with the movement of the explanatory variable. This does not mean that the model is 40 percent correct and a low R-squared does not mean that the relationship is not important

The two errors go in opposite directions and both are common. A high R-squared in a relationship without economic content is still meaningless and a low R-squared in a real effect is normal in finance where the returns of individual securities are dominated by things specific to the security. A factor that explains a small part of the variation in the returns of a single stock can still be worth a lot when applied to hundreds of positions

The standard error is the most ignored result that deserves the most attention. It indicates how much the slope would be expected to move if the same exercise were repeated on a different sample of the same process. A beta of 1.3 with a standard error of 0.4 and a beta of 1.3 with a standard error of 0.05 are not the same finding and the point estimate appears identical in both

The Line Is a Conditional Average

The fitted line does not predict what happens. It reports the average value of the dependent variable at each level of the explanatory variable within the sample provided

That distinction is the source of a lot of confusion when using regressions to make forecasts. If the beta is 1.3 and the market goes up 1 percent the regression claim is that the average result for this stock on comparable historical days was an increase of about 1.3 percent. It is not a claim that the stock will go up 1.3 percent and if R squared is 0.4 then most of what actually happens to the stock on a given day comes from somewhere.that the model never analyzed

There are two distinct uncertainties here and the software often reports only the first. One is the uncertainty about where the line sits which describes the standard error of the slope. The other is the spread of individual observations around the line which is much larger and governs any forecast about a single outcome. A model can pin down the average ratio fairly precisely and still say almost nothing about tomorrow

Why the Squares Matter

The method minimizes squared distances rather than distances and that choice has consequences that survive every result based on it

Squaring means that a point twice as far from the line contributes four times as much to minimizing the quantity. Therefore the fitted line is directed strongly toward the most extreme observations and a single unusual point can materially move a slope. In most data that is a technical nuisance. In financial data it is a structural problem because the extreme observations are the crisis days and those days have the greatest weight in the fit and behave less like the rest of the sample

The same squaring makes the method symmetrical. A point above the line and a point below it equally distant are treated as identical errors. Any question about the direction of the error and most risk questions are of that type is being answered by a tool that was created not to care

There is also a silent convention buried in the configuration. Minimizing vertical distances treats the explanatory variable as known and the dependent variable as the one containing the error. Swap the two variables and rerun and the line that returns is not the same. The arithmetic has assumed an address before any interpretation was applied

The Assumptions That Get Violated

Regression is based on assumptions and financial data routinely violates most of them

Linearity. The relationship is assumed to be direct. Many financial relationships are not particularly at the extremes which is exactly where they matter most

Constant variation. The dispersion around the line is assumed to be uniform. Financial data shows an accumulation of volatility so the dispersion is much wider in some periods

Independence. Observations are assumed to be unrelated to each other. Financial time series are autocorrelated

Stability. The relationship is assumed constant throughout the sample. Betas and factor loadings change over time sometimes abruptly

Violations Attack the Standard Errors First

Violations mainly cause standard errors to be underestimated meaning that relationships appear more statistically reliable than they are. That underestimation is the source of much confident nonsense

It is worth being specific about the mechanism because it explains why the damage is systematic and not random. The standard error is calculated under the assumption that each observation is new evidence. When observations are correlated with each other they are not. The sample still contains a thousand rows and contains far fewer than a thousand independent facts so the formula divides by a number larger than the actual information content and returns unearned confidence

Las observaciones superpuestas son la versión de esto que atrapa a personas que por lo demás son cuidadosas.Construir una serie de rendimientos de doce meses medidos cada mes significa que las observaciones consecutivas comparten once meses de datos.El recuento de filas parece generoso.La evidencia independiente es aproximadamente una doceava parte de ella y cada prueba de significancia calculada a partir de esa serie está inflada en consecuencia

The non-constant variation does something related. When the dispersion is much wider in some periods the fit is effectively dominated by the turbulent stretches and the reported uncertainty describes neither the calm nor the violent regime. Corrections for both problems exist and are standard in serious work. The failure is rarely due to a lack of available correction but rather the default output of each statistical package assumes that the assumptions hold and prints a t-statistic independently

Adding Variables Always Improves the Fit

R square cannot fall when another explanatory variable is added. The fitting procedure can always assign the new variable a coefficient of zero and reproduce the previous fit so any coefficient you actually choose should work at least as well. Therefore the improvement in R square is not evidence that the added variable belongs in the model

Adjusted R-squared exists to address this by imposing a penalty for each variable added and it can decrease. It is a partial solution rather than a solution because the penalty is a fixed formula and not a judgment about whether the variable has any business there

Push this far enough and the model is guaranteed to fit. With as many explanatory variables as observations the line passes exactly through each point and R squared reaches 1 which describes the arithmetic of the exercise more than anything about markets. Treat any regression with a large number of explanatory variables and a small sample adjusted for noise

Correlated explanatory variables cause a more subtle failure. When two variables move together the fit cannot clearly attribute the effect between them so individual coefficients become unstable while the overall fit remains strong. The symptom is a model that predicts reasonably and whose individual coefficients change sign or magnitude when the sample shifts slightly. Anyone who reads those coefficients as separate economic effects is reading noise

Correlation and Causation

A regression establishes that two series moved together in the sample. It does not establish direction and does not exclude a third variable that drives both

The financial version of this error is common. A factor that explains returns in historical data is interpreted as a determinant of returns. It may be a proxy for something else an artifact of the sample period or one of many relationships that appear when enough variables are tested

La dirección rara vez es tan obvia como parece.Hacer una regresión de la valoración de una empresa en función de su tasa de crecimiento y leer la pendiente como el valor del crecimiento ignora que las empresas caras pueden darse el lujo de invertir más lo que aumenta el crecimiento.Ambas historias se ajustan de manera idéntica a la misma dispersión de puntos y la regresión no puede distinguirlos porque nunca fue diseñada para hacerlo.Elegir entre ellos requiere una discusión sobre cómo funciona el mundo hecha por una persona antes de que el resultado signifique algo

A Coefficient That Was Selected Is Worth Less

Run enough regressions and some will produce impressive statistics by pure chance. With a conventional significance threshold of 5 percent one in twenty pure noise tests will appear significant

Modern data availability makes it possible to test thousands of relationships quickly. Published research in finance has documented a large number of factors that supposedly explain returns and a substantial portion is not replicated out of sample

The regression-specific form of this is specification search. The same underlying question can be asked with different control variables different sample windows different definitions of the dependent variable and different treatments of outliers

The fix is ​​to require an economic reason for the relationship before accepting the statistics and to require that the relationship contain data that was not used to find it

Beta in Practice

Beta is the regression that every finance student encounters first and clearly demonstrates the estimation problem because the answer depends on options that are rarely reported along with it

Which index serves as the market changes the outcome. The same goes for the return frequency since a beta estimated with daily data and one estimated with monthly data over the same period will be different and for less liquid securities the daily estimate creeps toward zero because the price has not moved yet and not because the stock is unresponsive. The same goes for the window length which changes a longer sample by the assumption that the company has not changed

Then there is stability. A company that issues a large amount of debt sells a division or shifts its customer base has genuinely changed its sensitivity to the market and a beta averaged across the transition describes a company that no longer exists. Check whether the relationship is stable between subperiods since a coefficient averaged over a period containing a structural break does not describe either regime

Plot the Data Before Trusting the Summary

Plot the data instead of relying on summary statistics because very different data sets can produce identical regression results

A clean linear relationship a curved relationship a flat cloud with an end point dragging the line and two separate groups with a line drawn between them can return the same slope the same R squared and the same standard error. Summary statistics compress the data to a few numbers and the compression discards exactly the features that would tell you that the model is wrong. The image takes a few seconds and is the cheapest diagnosis available

Therefore the questions worth asking for any regression are quite mundane. What is the standard error not just the coefficient? How many variables were tested before this specification. Are the observations independent or overlapping? Does the relationship hold in the first half of the sample and in the second? And is there any reason expressed in terms of behavior structure or constraint why this line should exist?

The Bottom Line

Regression estimates how one variable has moved with another and provides uncertainty around that estimate. It supports beta and factor models and is misused when the result is read as causality when violated assumptions make significance appear stronger than it is or when the relationship was discovered by testing many possibilities. Demand an economic explanation and out-of-sample confirmation before believing any of it. The fitted line is an average taken from a sample that has already happened and will be produced in the same format.clear whether or not there is something to find