Disconnect between Reverse Engineer and Ranking System Performance Tools?

Research
1

I have a “simple” ranking system (generified factor names):

Note I am ranking formulas “A” and “B”, as well as ranking system “DisconnectBetweenRevEngAndPerformance” as “Higher” valued.

Now I investigate with Reverse Engineering. Weekly price performance relative to S&P 500:

[size=3]–> +30% results:[/size]

[size=3]–> -30% results:[/size]

So to recap, the overall ranking system (“DisconnectBetweenRevEngAndPerformance+”) shows:
[size=3]+30%: Avg Rank=44.54 StdDev=28.20[/size]
[size=3]-30%: Avg Rank=59.57 StdDev=24.31[/size]

So the +30% results have a much lower average rank (44.54) vs the -30% results (59.57). So I would expect to need to reverse the ranking system method - Lower values ranked more highly.

However, when I run the Ranking Performance Tool, I see that Higher ranked results give best performance:

(The results page doesn’t show it here, but take my word for it - the overall ranking system is still ranked as “Higher”)

Note this is for the same period, both set to weekly evaluation/rebalancing.

Any ideas what could be going on here?

2

This was posted a long time ago, and the OP has likely moved on. But I just tried the reverse engineering tool for the first time, and I find myself stumbling over the same issue: A ranking factor (a) assigns higher ranks to higher factor values and (b) “works”, i.e. higher factor values are associated with higher future stock gains, yet, using the reverse engineering tool I find that the average rank for stocks outperforming the SPX by 10%+ is lower than the average rank of those stocks underperforming the SPX by 10% or worse.

If you

  • see the flaw in our understanding, or
  • can point me to a thorough explanation of the tool, or
  • can confirm that the tool isn’t working as designed/expected,
    I’d really appreciate the help.

Thanks,
Matt

3

Matt, I can confirm seeing the same apparent concern. The only description that I have found of this reverse engineering process is displayed by clicking on the question mark in the displayed page. I believe the relative SP500 performance choices should be considered inverted, i.e. for the +20% choice, the S&P500 outperforms the stocks by 20% rather than the stocks outperforming the s&P500. Likely just a comparison operation error, but if that is not the case then it would really help for P123 staff to say so.

4

Bob, thank you for your thoughts. Your explanation seems entirely plausible and I definitely second your request for P123 staff to chime in to confirm/reject it.

Judging by the few available posts about the reverse engineering tool, it seems that so far it has created more confusion than excitement. I think that’s a shame, because the intuition behind it is compelling. With a little more documentation and the fix you suggest, this could be a powerful tool.

5

I must admit to only skimming the reverse engineering post here…but I will add what I can recall from using it and why it seemed strange.

There is a difference between outliers and average factor performance. Suppose that only extremely high beta stocks return 50% over a 2 week window. Also suppose that only 1% of the high beta stocks achieve this while the other 99 greatly underperform by 5% during the same 2 week window. And suppose that low beta stocks never return 50% in a 2 week window but they outperform on average by 5%.

So you run your scan for stocks that jump 50% or more in 2 weeks. You see that they are high beta stocks. So you add this as a factor and your ranking system sucks. And when you go after low beta stocks, it improves it. All this despite your highest earning stocks all being high beta.

But maybe I am barking up the wrong tree. Just wanted to add my nickel (we no longer have the penny in Canada).

6

Hey Kurtis, I understand your example but am not yet completely sold on it as the complete answer. I did, however, create a test case that proves my assumption of inverted S&P500 performance comparison logic is wrong. But the test case doesn’t provide an obviously clear picture for me either.

I am still checking and analyzing, so more later!

7

I have some results from a test, some conclusions, and some unresolved concerns.

To simplify checking the reverse engineering results, I created a test scenario. Five tickers were chosen as of 11/25/2017. Each of them represented a different market cap classification (nano, micro, small, mid, and large cap). Their returns from 11/25/2017 to 2/24/2018 in percent were 150.0, 58.97, 29.29, 10.47, and -23.03. SPY managed 7.34%. Note that 4 of the 5 stocks outperformed SPY. The ticker list was Ticker(“BSTG,PTI,ABCD,AB,EIX”) (listed from small to large market cap).

Using a ranking system with one factor, MktCap, lower is better, with a custom universe that specified only those five stocks, the performance for five buckets from 11/25/2017 to 2/24/2018 with 13 week rebalancing looked as expected, fantastic evidence that for these conditions, smaller is better. The generated ranks for 11/25/2017 for the stocks in the ticker list above were 80, 60, 40, 20, 0. Using reverse engineering and specifying the five tickers in the screen command line, with a 3 month rebalance and the same dates gave the following:

SPY+30%: Avg. Rank 86.76, Std. Dev. 12.63, 2 values used
SPY+20%: Avg. Rank 77.30, Std. Dev. 18.65, 3 values used
SPY+10%: Avg. Rank 77.30, Std. Dev. 18.65, 3 values used
SPY-10%: Avg. Rank 17.10, Std. Dev. 15.77, 2 values used
SPY-20%: Avg. Rank 17.10, Std. Dev. 15.77, 2 values used
SPY-30%: Avg. Rank 30.87, Std. Dev. 26.32, 3 values used

The SPY+ reverse engineering results are as expected (higher ranking performed better in general, and the higher the performance the fewer stocks). The SPY-10, -20, and -30% average ranks reported do not make sense because there was only one low (0) ranked stock (EIX) that performed worse than SPY. I believe the SPY-10, -20, and -30% average rank should report zero, with zero standard deviation, and 1 value used (to match the one stock with a negative return and also a return worse than SPY).

I don’t know what else to attempt at this time. Any suggestions or comments???

8

I hope P123 chimes in on this one.

Walter

9

I would like to know how the reverse engineer works.
It seems to be demanded within the P123 community.
Thanks in advance.