I have read several threads on out-of-sample performance of portfolios, mainly on R2Gs as their performance is accessible to everyone.
As there seems to be a common understanding of the importance of oos performance, I was wondering if there is a way to retrieve more oos data from R2G portfolios (besides ranking them by oos return and months launched).
Common shortcomings like chosing the “wrong” benchmark and “living in a bull market since the launch of R2Gs” are known, but as we progress and more time goes by, I was wondering if more R2G ranking factors could be restricted to oos data.
I would like to see an additional option on the R2G main page, that lets you sort R2Gs by oos data.
That way, the validity could be more easily assessed (provided the R2G has been launched some time ago).
This might not make the top of the list of “feature requests”, but might be interesting as we see more R2Gs being launched and have more oos data.
I’d like to see OOS Sharpe and/or Sortino as part of the ranking criteria. I’ll also take this opportunity to once again complain about how the rankings are calculated for “low risk”. “out of sample”, “performance” and “income”.
Oos Sharpe and/or Sortino were actually two of the main factors I am missing (as oos) as well.
I wrote that down in my first post, too, but must have deleted it before submitting the post
Might be worth a feature request if more people are interested.
However, P123’s time and money is certainly limited (generally speaking) and I prefer that limited working power to be directed towards actually designing new/more features (instead of working on how data is presented, like oos R2G ranking criteria).
But in the long run more oos ranking criteria might be helpful for both designers to be able to efficiently present their models and for subs to be able to find models based on their true preferences.
Personally getting boring to construct my own criteria along “low risk”, “out of sample”, “performance” etc. Would prefer to have my ouw saved custom criterias.
As well would like to see my simulated and live books among R2Gs list, partly due to comparing with R2Gs, but mostly to compare my books (live and simulated) with more criteria than available at the moment.
As for criterias would like to see DD recovery stats and trailing 1 month, 1 quarter, 1 year DD performance stats. Think this is interesting for whose seeking the comfort investment running.
Konstantin - drawdown recovery stats could be a dangerous thing particularly if it is in sample. It gives the investor justification for investing in very volatile systems with the assumption that losses will be recovered. In real life the system may not recover.
Steve, I think every stat could be dangerous then misunderstood and abused. I see your point and agree this is dangerous, but again this is as much dengerous as MaxDD, it can always double and we know this
Personally need this for quite resilient book and for sure will not use the only stat. I found it very difficult and not descriptive current book simulation stats (I mean list table) and would like to have ranking at least comparable to R2Gs have.
tkp - for me this stat is worse than others because to rank hi the system would first need to have a high drawdown, the higher drawdown the better. That way the recovery can be that much better. Anyways, I don’t really care so long as this parameter doesn’t make it into one of the canned sort filters.
It depends on how knowledgeable the average R2G sub is.
Personally, I must admit that recovery stats would not be my first choice either.
I don’t see how they add much value to Sortino and MaxDD in the first place - risk-adjusted return and loss aversion would be of prime interest anyway, right?
If high Sortino and low MaxDD are achieved, there’s no need for recovery.
So, recovery stats might add more information, but they might be only of secondary concern and only add value (in terms of information) in very special situations (after a drawdown occured - how would you define the average drawdown?). So it might be helpful for private ports but maybe more difficult and confusing for R2Gs and should therefore be left out of the predefined sort filters.
Steve, fips - you are right. I was thinking of DD recovery exceptionally in Book modeling, so I don’t care of R2Gs. Have confused you as put different futures proposals into one place. For sure DD recovery is criteria of not models selection, but book running. Constructing book before run I’d like to know what to expect in downturns, like how deep (MaxDD) and how long (DD recovery) I can more or less trust to book/system and is it time to start to worry. Anyway agree this is secondary. But this secondary comes very quickly, just after book goes live and first DD come.
First lets get more in-sample data. This is crucial since this year I saw many competing R2Gs not behaving like they should have. Adding more years will make the historic hypothetical studies more reliable, particularly for small portfolios. We should have 90’s and 80’s at least. I would be willing to pay much more for this addition.
I just skimmed through the first ten pages of the feature suggestions and used the search bar - no luck in finding a feature suggestion regarding more in-sample data.
If there is one, I would be happy if someone could provide the link. Otherwise, I would be like to start a new request.
I would not be holding my breath in the hopes of getting musch “useful” fundamental data prior to 1999. Some of the most powerful factors for ranking systems were not avaialbe prior to 1999 (at least not for a reasonable price). For example, many ranking systems use some form of Earnings Estimates either directly or in the form of future value formulas (like Forward P/E ratios) but my understanding is this type of data just started to be collected in 1999 and 2000. Price, volume, book value, eps, etc, do go back farther, but many powerful powerful ones do not.
Years ago, I paid a significant price to get price and volume data on delisted stocks prior to 2000 and was disappointed to find that most small cap price data was not available prior to 1992-93. Sure the data provider said they had data back to the early 1980s but that turn out to only be for large cap stocks.
COMPUSTAT does have it o806. Also you can still test many aspects of your systems even if you cant use estimates. Many of my systems don’t use them at all. I hear over and over again how robustness is important and yet I don’t see much of a move to expand the historic simulation period even though it is the best thing we could do at the moment in terms of improving robustness across all systems. I know it would be expensive but I am willing to pay more myself. Keep in mind academic studies are often multi-decade in span.
I am not sure if longer data periods (beyond what we have now) is “the best thing to improve robusteness”.
There are quite a number of other things that you can do to check for “robusteness” (varying your variables systematically, decrease the number of variables etc.).
Of course, longer data is always better than shorter but what we have now is sufficient for me.
Resources are spent wiser for other things that members urgently want, imho.
Werner
I agree with you. I know many long short professionals and they use longer data sets. But, this has been talked about at length in the past. I assume since it’s gone nowhere it’s cost prohibitive.
We all agree that the information is valuable.
However, we can’t discuss much further unless we know how much time and money it takes to make earlier data available.
I agree with Tom that this has been discussed as costly before, but I cannot pinpoint the exact thread.
One guy I know started a hedge fund, then spent 3 years cleaning the data. His data only goes back to 1990. So…not sure how much it costs, but I know he had a small team from MIT cleaning data for 3 years.
