There’s been a lot of unrest about simulations generating different results.
It’s true. There have been algorithm changes in some ratios (see posts that start with IMPORTANT), there have been bug fixes, and there have been data fixes in Compustat too. Since we do not snapshot the majority (90%+) of the point-in-time ratios, rather we re-calculate them on the fly using the latest versions of the data, simulated results ran in the past change when run today. There are arguments for and against on-the-fly ratios, but we are sticking to on-the-fly. It is the only manageable way to do it, so this problem with changing sims will persist. I’ll come back to how we’ll address this…
I have noticed another disturbing trend: out-of-sample results are being elevated to gospel.
This is just not so. Out-of-sample results, just like simulations, are nothing more than possible outcomes of a strategy. Lets remember what we are doing: constructing a portfolio of 10 stocks where the decimal precision of 0.01 in the rank determines the order. On top of that ranks are based on relative orders of financial data full of N/A, pre-announcements, and outliers. It’s no wonder that any small perturbation can cause the top 10 to be completely different.
In other words: a ranking system’s worth can only be judged on large groups of ranks, deciles or even quintiles. But systems of a handful of positions are being created in P123 using fractions of 1% rank! It’s absurd. And on top of that , layers upon layers of buy & sell rules are applied, and market timing to boot. This makes falling into the curve fitting trap very easy and natural. I’ll go as far as saying that engraving past ratios in stone would make curve-fitting even worse.
So what to do? It’s our presentation that needs to change. Simulations will no longer be presented the way they are now. Rather as a multitude of 3 year simulations with varying start dates and random noise. The true worth of a trading system is not the past results (whether sim or “live” it’s indifferent, it’s the past), rather as a probability of outperforming a benchmark in a three year period by x%. Or something like that. No more “Portfolios”. We should also rename Portfolio123 to System123
Thanks