We can’t – but Mr. Market and Father Time can and will do so loudly and boldly.
This has been hotly debated in the forums pretty much since R2G was launched almost three years ago, and there is still no consensus answer (as Marco noted, we hoed rolling tests would be revealing but it did not turn out that way).
I think the problem is caused by the fact that the things about which we complain really have nothing to do with what steps a designer took or did’t take or even whether a model is “robust” or not. It has to do with intention. Is somebody intending to design a model that succeeded in a specific and closed environment (the sample period)? Or is somebody designing a model with the intent to perform in the unknowable future?
If one intends to design something that succeeded in a specifically defined and closed-ended sample, sound quant processes can be and have often been shown to have been very successful. So, too, can design based on investment principles.
If one intends to design something that has a reasonable probability of succeeding more often than not in the unknowable future, then exclusive reliance on quant processes cannot work; one can get lucky if and for as long as the future more or less replicates key aspects of the sample period. But over time and as the world evolves, that sort of thing cannot be sustained.
There are four ways to measure designer intent:
One is to reveal the trading systems, but for obvious reasons, that is an absolute no-go.
Two is to consider designer-written descriptions. This can be as effective or ineffective as designers choose to make it.
Three is to try to objectively measure stylistic choices, as we have done with the style ratings (actually, these don’t measure designer intent; they measure what the designer achieved whether or not it was intended).
Four is to rely on the team of Mr. Market and Father Time. The good part is that as time passes, this is foolproof. The market will do what it will do and whoever does what Mr. Market wants will be rewarded and we’ll objectively know how well models stand. The bad part is that it may require patience for new models. But that’s the way the world is. New asset managers, mutual funds, ETFs, quant boutiques, etc. have to tough ti out waiting to accumulate live track records. Newbies in every walk of life need time to show what they can do. Our current platform acknowledges that we have to live in this world; we can’t change the world to get outcomes as quickly as we wish.
Our old sim-oriented presentations kept the over optimization, curve fitting etc. conversations alive far longer than should have been the case. As time passes, and we live with the new approach, I believe we’ll see that this line of conversation will diminish and eventually vanish from Smart Alpha related discussions. (It will likely persist, but in other threads relating to how one should approach strategy design, which is where that topic should be discussed).