Out of Sample Period

Hi all,

One of the things I was excited about the switch to Factset is the additional 5 years of backtest (from 1999 to 2003)

I was wondering, would you consider those additional years as an out of sample period for your models?

My models were backtested with 15y of Compustat data and they perform well in the additional 5y, is this relevant from your pov?

Thanks,

Alberto

This is indeed an out-of-sample period. If you haven’t run any tests on a period, it’s out-of-sample by definition.

However, it’s somewhat problematic for several reasons. First, there’s some missing data. A few of our functions have few data points back then. Second, the period immediately after the dot-com crash was a golden one for small-cap value stocks. In general, most of the factors we use performed better against almost all benchmarks in those years than in the post-GFC years. So the likelihood of your model outperforming is higher during that period than in, say, the 2017 to 2019 period. Third, out-of-sample performance that’s fifteen to twenty years old is useful for validating that your model isn’t curve-fit, but will tell you very little about out-of-sample performance in the future because conditions back then are so different from conditions now. Remember that in order to buy a stock you had to call your broker and place an order. Remember that interest rates on one-year T-bills were between 2% and 6%. Remember that the sectors that really outperformed back then were real estate and materials, and the sectors that performed worst were technology and telecom. Remember that few people were using tons of data to sift through and analyze stocks, and that (unless I’m mistaken) not a single university offered a course in valuation. In short, we’re living in a very different world when it comes to the stock market. We now have the tools and factors and formulas to go back and see what we could have done back then to make a killing. But what tools will people be using in fifteen years to discover how we could have made a killing in 2020 and 2021?

Many factors seemed, to me, to be super-charged in the early 2000s. I don’t know the right approach, but factor erosion seems a reasonable hypothesis, and I suspect most multifactor models will have better alpha from 2000-2010 than from 2010-2020.

Thanks Yuval. Very insightful as always.

I remember having read in the forum the incredible small and micro caps performance in the early 2000s as SpacemanJones mention. It seems anything small and illiquid did amazing back then, apply factors and then it is out of this world.
I won’t take the backtested returns over that period as meaningful to the future performance of the systems. I am just happy the model did not break, although looks like there’s not too much merit to it.

Now, that opens the eternal debate of “is there any backtest relevant at all”? I believe backtests are tremendously useful, the danger is when people expect the exact same results/equity curve in the live trading, which can happen (if you stick to it in the long run mostly), but likely won’t, not because it is a bad backtest but because the environment changes rapidly.

It is not just small caps which did well.
Here is the out-of-sample performance 2000-2005 of a model which targets Russell 1000 large caps. The model was designed for period 2005 to 2020 and it also performed well over the 5 years starting Jan-2000.


Got it. Factors, as we know them today, outperformed greatly during 1999/2005 period.
So, we should focus on the recent 5 or 10 years as a better proxy of what comes next… is not that called “recency bias”?

2000-2003 was a golden period of value investing. Buffett rebuilt his reputation during this time after being ridiculed during the dot com bubble. Small cap value in particular did extremely well. And of course, value has struggled mightily in recent years (as least in regards to traditional metrics). Do I think we’ll see a replication of that era again in the future? Sure, just give me a heads up when.

Yes, it’s called “recency bias,” but you should think for yourself and come to your own conclusions. Is ignoring recent events better or worse than focusing only on them? Is it more probable that the next two or three years will resemble some of the last ten or twelve years or that they’ll resemble a period before that? Will you get a more well-rounded system if you look at a longer period (ten years or more) than a short period (five years or less)? When in doubt, I say, take the middle road.

Thanks a lot fr the advice Yuval.