There’s a very funny article here:
In it, some researchers created 2 million fundamental quant strategies using the Compustat codes and tested them all. The ones that worked best are incredibly nonsensical.
There are some serious methodological problems with the study, but it’s good for a laugh.
“Methodological problems”? What are they?
First, “they form portfolios using a one-dimensional sort on each of the variables. The portfolios are rebalanced annually, creating long/short portfolios that go long the top decile on each measure, and short the bottom decile.” The period is 1972 to 2015, so basically they’re running 43 tests (one per year) of two million long/short portfolios consisting of two deciles, one long and one short. 43 tests is far too few tests when you’re dealing with that amount of data–43 years’ worth and, I’d guess, at least 2000 stocks per year. A large percentage of your results are going to be outliers.
Second, they’re looking for the effects of ONE ratio. If such a ratio existed, everybody and their grandmother would be using it to make millions of dollars. We here at P123 know that there’s no such thing as an effective one-factor ranking system.
Third, their method of creating factors makes no sense because nobody in their right mind would go about doing it this way.
Fourth, they’re using 156 factors and they say they are computing all combinations of growth rates, ratios of two, and ratios of three. I find it impossible to believe that all such possible combinations would add up to only 2 million. In fact, the number of permutations of 3 out of 156 possible factors alone is close to 4 million, and that’s not counting the growth permutations. And they’re not entertaining the possibility of factors that require a lot more than 3 Compustat codes or that combine ratios with growth.
[quote]
A large percentage of your results are going to be outliers.
[/quote]What’s wrong with outliers? Isn’t the point of picking stocks to find the outliers?
[quote]
Third, their method of creating factors makes no sense because nobody in their right mind would go about doing it this way.
[/quote]How would you program a computer to iterate through all possibilities?
[quote]
Fourth, they’re using 156 factors and they say they are computing all combinations of growth rates, ratios of two, and ratios of three. I find it impossible to believe that all such possible combinations would add up to only 2 million. In fact, the number of permutations of 3 out of 156 possible factors alone is close to 4 million, and that’s not counting the growth permutations. And they’re not entertaining the possibility of factors that require a lot more than 3 Compustat codes or that combine ratios with growth.
[/quote]Good catch!
[quote]
Permutation takes into account the order of selection: in high school we were putting “permutation locks” on our lockers. The order of the numbers we entered into our “permutation locks” mattered.
The number of combinations—where the order does not matter–for 3 items out of 156 is 620,620 (COMBIN in Excel).
Thank you for the link.
-Jim
[quote]
You could put it that way. What I meant was that using such a tiny number of tests over such a huge amount of data is going to inevitably result in many iterations that exceed the norm purely on the basis of chance selection. Rolling backtests are needed here.
[quote]
I would limit the computer to looking at data points with some financial relationship to other data points. Each data point would have a definition that explained how it was computed from the raw numbers that the company’s CFO was working from. The computer would then draw a map between data points based on the number of relationships between them. Depreciation and cap ex, for example, would be very closely related since they rely on many of the same raw numbers, while depreciation and the number of preferred shares would not. Formulae could then be based only on data points that had some relationship to each other, together with value formulae (data points with some conceivable relationship to the company’s ability to make money compared with the company’s share price or net worth). I’m not a computer programmer and I think the whole project of “finding anomalies” is garbage, but you asked.
(And if you wonder why I think “finding anomalies” is garbage, it’s because the idea of anomalies is based on exceptions to a random walk or a perfectly priced market, while I think of the market as a badly oiled Rube Goldberg machine with little randomness and very imperfect pricing that runs on a bunch of sound and unsound financial and behavioral principles that are ripe to be taken advantage of.)
Boy is this an excellent point!!! A point that the authors seem to agree with.
See image for strong point of agreement (Wes Gray quoting the article). Both Wes Gray and the authors agree with this.
And the authors tried to account for this in their own humble way (second image)
-Jim
I don’t think the authors acknowledged that the tiny number of tests per strategy were responsible for their crazy results. Or that using rolling backtests would have been a much more robust method.
