Ranking Decile Performance does not equal Screen

I have constructed a short ranking system which has a 90th decile performance for the last three years of about 17%. (https://www.portfolio123.com/rank_perf_show.jsp?rankid=306805).

I have also created a screen (https://www.portfolio123.com/app/screen/summary/181866?st=0&mt=1) which I believe should equal the performance of the ranking system, but the results are quite different. The screen has a performance of 8.93% vs ~17% for the ranking system.

In the screen, I have made the slippage 0% and the carry costs 0% so that the screen should equal the ranking system. I must be doing something wrong and would appreciate some help as to what I am missing.

Many thanks

Paul


You would need to set the number of stocks to be about equal to 10% of the universe (all fundamentals selling for over $3). You are currently backtesting the whole universe - I think.

I think I have set that already. In the screen rules, I have:

close(0)>3; and
rank > 95

I believe that this should give me the same 95th-100th decile performance of the ranking system.

Is your link pointing to the right screen?

Walter

The screen defaults to “buy at the next open”. The RS tester “buys” at the next close. It is probably not the issue (buying at the open usually–but not always–gives better results), but it may be worth a try.

I have just checked, it is pointing to the right screen.

Chipper6, if screen defaults to next open which is usually better than next close, I would expect to see a better result from the screen than the ranking performance. However, I am seeing the opposite.

One likely minor adjustment to check. Try Rank>=95 in the screen. That should equal the top 5%, whereas Rank>95 should give the top 4%. There might be a few extreme winners at the 95th percentile in the backtest. And since this is a short screen, shorting at next close would reverse the normally worse screening results versus next open, if Chipper6 was referring to the effect on a long model.

Thanks for the suggestion. I made the change but it did not make any noticeable difference.

I had something similar happen once. Can’t recall exact details but it had something to do with the universe having a massive amount of NAs for a certain variable and it was handled differently in the rank test vs. screen. I had to make a custom universe to filter out the NAs and then it ran as intended. Again, this is probably not a lot of help unless I can find the support e-mail. I will look.

I have changed my opinion on this one. Something with the short ranking is off. I ran it with just 2 price based variables so there should be no NAs. When I did the annualized return ranking it said the top bucket was 40% annual return. When I switched the chart to show the time series bucket returns there was no such thing.

I have had this problem for a number of screens in the past. Most of the time the rankings are at least directionally correct, but occasionally they are way off. I have changed the timing, open vs. close and experimented with a number of other parameters, but I can’t solve it either.

[quote]
In the screen rules, I have:

close(0)>3; and
rank > 95

I believe that this should give me the same 95th-100th decile performance of the ranking system.
[/quote]I would create a custom universe as hemmerling has suggested. If your ranking system has more than one factor, it matters. It’s not just the NAs.

Steve Auger (stockmarketstudent) posted about this some time ago in a different thread.

I only use custom universes and I still have the same problem. The other thing I noticed is the decile rankings are almost always higher.

[quote]
I only use custom universes and I still have the same problem. The other thing I noticed is the decile rankings are almost always higher.
[/quote]I was addressing Paul Dyson. His screen had the rule “Close(0) > 3”, which practically guarantees that his screen will have different results than his ranking system.

Mike, what exactly are you doing, and what discrepancy are you seeing?

Chipper6, thanks for your suggestions.

I am confused, though. Why, if the screen has a rule “Close (0)> 3” does it guarantee that the screen will not match the ranking system? In the ranking system, I have the filters applied after ranking set with a min price of 3.0. My understanding was that by putting “Close(0)>3” in the screen, it would equal the ranking system.

I have spoken with Paul DeMartino from Portfolio123 and they don’t understand why my ranking system and screen are showing different results and are investigating.

Chipper6 and pdw,

Typically when I run the ranking I break it into 100 bins to look at percent ranks. When I find one that looks promising, I try to duplicate that using the rolling backtest using the same custom universe cutting off the top say 1%. The rolling backtest almost always underperforms. The more complicated the ranking, the greater the underperformance tends to be.

As an experiment, I did this last night for a custom ETF universe using only 26 week relative strength in the ranking system. The top 1% had annualized returns of 14.9%. When I do the same using the rating>=99%, the annualized return is about 11.5%. I used a 5 year window and weekly rebalancing.

I made all of this stuff public so others can take a look. Not sure the links will work right, but below is where I show them.

https://www.portfolio123.com/app/ranking-system/306246

https://www.portfolio123.com/app/screen/summary/115759?st=4&mt=9

@mdimillo77,
I can’t say I’m too familiar with how ranking works on the short side of things, but I had run into a similar issue a while back with regular long rankings. It mostly had to do with the treatment of NAs, as I remember. I trapped the error by duplicating the ranking system in the screen, while also duplicating P123’s methodology for ranking systems through a series of “FRank()” and “FRank(Avg(@rank1,@rank2,…,@rankN),#Previous,#DESC)”. I then backtested each quantile through the screen until the results were more or less the same.

Read more here: https://www.portfolio123.com/doc/RankingDoc.pdf.

It’s hard for me to troubleshoot your issue since your systems reference custom variables which I cannot access. However, I would suggest that you compare the ticker-by-ticker rankings on the screens versus ranking system, paying particular attention to the top and bottom quantiles. That would at least let us know if we can rule out ranking issues as the major source of the discrepancy.

Good luck!

[quote]
Chipper6, thanks for your suggestions.

I am confused, though. Why, if the screen has a rule “Close (0)> 3” does it guarantee that the screen will not match the ranking system? In the ranking system, I have the filters applied after ranking set with a min price of 3.0. My understanding was that by putting “Close(0)>3” in the screen, it would equal the ranking system.

I have spoken with Paul DeMartino from Portfolio123 and they don’t understand why my ranking system and screen are showing different results and are investigating.
[/quote]It’s complicated.

It has to do with the fact that a screen first ranks the stocks in the universe and later applies the rules, but the ranking system backtester applies the price>3 rule before ranking (which means that it treats price as if it’s part of the given universe)

When you have a single factor ranking system, you will end up with gaps; so that RankPos <= 10 may come up with less than 10 stocks.

With a multiple factor rs, the ranking order will also be affected.

Chipper, Thanks for the reply. I totally hear you on the stock universes. I use the same dollar cutoff on the custom universe and the ranker. The example I shared is different. There is no price rule when you just use ETFs. That is why I used it as an example because it is the most basic universe possible and there aren’t any limitations.

Primus, I agree on the NAs. I think Kurt pointed that out earlier. I tried to share my ranking system as a public system, but I guess I messed up. The only variable in my system is 26 week relative strength. There shouldn’t be any NAs. How can I compare ticker by tickers of the ranking system? It’s easy to do on the rolling backtest, but I don’t know how to do that on the ranking percentiles.