Ranking factors with high return in most high buckets, except the topmost buckets

Dear all,

When performing single ranking factor tests using the standard P123 Ranking System Performance Test with 20 buckets, some ranking factors have good performance (high return) in most of the top buckets, except in the topmost bucket (95-100). Others perform well in the 70-90 buckets but the return falls drastically in the 90-100 buckets. I’m performing 16 year backtests using the following universe: Close(0) > 1, Vol > 100K, MktCap > 100M.

I’m trying to find out the reason for this and an effective way to exclude the topmost 1 or 2 buckets from the ranking factor (“trim” the results).

Some theories for why this happens are:

  • The top performing stocks in well known factors get excessive attention and this causes them to be overpriced;
  • The top performing stocks in fundamental and value indicators have “too good to be true” results and are possibly manipulating their earnings.

What do you think?

Do you know of any effective way to "discard the top 5 or 10% of a single ranking factor (node) in a ranking system with multiple factors?

Thanks in advance.
Bruno


example-bar-graph.png


QuickRatioQ.png


AltmanZPriv.png


FCFQ.png


MACDD(6,13,5).png

Hi Bruno,

The magic of a good ranking system is to find a few factors that work well in combination. If you succeed in it, you will see the highest ranked stocks outperforming by a large degree, such as in the below example. This eliminates the need to excludes the highest ranked stocks that would fail in a single-factor ranking system as illustrated in your examples.

Simulations based on ranking only - without additional buy rules - are in my opinion the least likely exposed to curve-fitting.

Happy ranking!


Bruno,
I think it is easiest to use Frank in your custom universe.

So, if you check the performance with 100 buckets for 1/Pr2BookQ and the top bucket performs badly put Frank(“1/Pr2BookQ < 99”) as a rule in your universe. The top bucket will now be your best performing bucket for that universe and you can see how this universe works in your sim.

Regards,

Jim

Hi Bruno,

In addition to filtering with FRank, you may want to try ZScore, too. Something like Between(ZScore(“Pr2BookQ”,#All,1.0),-3,3). Some of the parameters may need to be tweaked.

Best,

Walter

EDIT: It may also be worthwhile to examine factor distribution under Data Views. The only problem I have wth it is there seems to be no way to specify a Universe to work against.

Bruno,
For a background on why certain factors seem to work better for certain universes, I would point you to the books under the Factor Study section at:
https://www.portfolio123.com/doc/books.jsp

sevensisters: I don’t use any buy rules on my portfolios and only sell a stock if it drops below a certain ranking threshold (eg: Rank < 95). The threshold depends on the ranking system used in the portfolio. Maybe I am worriying too much about this and the other factors in the ranking system will balance out these abnormalities.

Jrinne and wwasilev: That’s good info. I’ll test all of your suggestions. BTW, I didn’t know about the “Factor Distribution” tool, it will be very useful.

davidbv: Will buy a couple of those books.

Thanks everyone for your help and good advice.

@Jrinne:
I applied your FRank suggestion on the following factor formula “avgvol(10)/avgvol(120)” sorted by higher rank=better. Without any FRank, I get a nice distribution with 30% at the high end (see below).
If I apply FRank to eliminate the uppermost bucket as FRank(“avgvol(10)/avgvol(120)<99.5”) I end up with mostly empty buckets (see below). What am I doing wrong?

@wwasilev:
If I were to use ZScore instead, how can I figure out what a typical range of values is for the uppermost bucket?

Thanks in advance!


vol_no_FRank.jpg


vol_with_FRank.jpg

Florian,
Here is what I got from your idea. avgvol(10)/avgvol(120) is the only stock formula in the ranking system.

Pretty sweet ranking performance!



Test of Frank Rank.PNG

Florian,

Thanks for your post. Even though my performance looks different than yours, I am not sure FRank is really restricting the universe for me. I tried FRank(“avgvol(10)/avgvol(120)<99.5”) down to FRank(“avgvol(10)/avgvol(120)<80”) and the performance graph did not seem to change (200 buckets).

I will keep looking into this.

Bruno,

Your charts show that the top bucket has lower returns than the next bucket, but what is happening with the stocks that you are actually trying to buy?

Since you don’t use other buy rules, your Sims & Ports will always buy the very highest ranked stocks. Therefore, the use of 20 buckets in the ranking system performance test doesn’t show you very much about the stocks that you will be buying. I suggest that you use 200 buckets instead to see the performance of the stocks that you will actually be buying.

For example, if you have 2000 stocks that pass your universe then you will have 10 stocks in the top bucket. As your Sim runs, if it sells 2 stocks that are the lowest ranked, it will buy the 2 highest ranked that it doesn’t already own from the 10 in the highest bucket.

You will never buy stocks from the lower buckets unless you have additional buy rules that force the Sim lower in the ranks to find stocks that meet all the buy rules. The lower buckets do give you additional information about the ability of the ranking system to discriminate the best stocks, but you won’t be buying any of them.

Florian,

You did get 30% in one of the highest bucket, but you only got 5% in the top (green) bucket. We must look at the stocks that our Sims will actually be buying.

Jim,

If you run 200 buckets the top bucket is way down. :frowning:

Zscore returns the number of standard deviations (SD) a value is from the mean. However, to be most useful, the random variable should follow a normal distribution (ND). If you want to assume a ND, then pick a SD from the attached diagram.

I’ve seen a transform to convert non-NDs into approximately ND. I’ll have to look in my library for the transform. I’ve never tried it but maybe it could be useful here.

Best,

Walter


Gents,

although this factor is the only one in the ranking system (with 100% weight), I still get most of my buckets empty. This should not be the case, because my custom universe includes 2800 stocks as of today. Any idea where the problem might come from?

@Walter:
Thanks for the ZScore illustration. Reminds me of the IQ distribution when my little son went to the child psychologist; he is between +2 and +3 sigma, so I look forward to see him one day applying his brain to P123 :slight_smile:

sevensisters,

your FRank formula has an error. It has to be FRank(“avgvol(10)/avgvol(120)”)<99.5 .

Matthias

Thanks Matthias! Works with your correct code. The top bucket can be made to be the highest bucket. The universe rule I used is actually: FRank(“avgvol(10)/avgvol(120)”)<94.5 for this rank performance.

BTW, Assuming there are no data errors, isn’t this highly statistically-significant proof that the efficient-market hypothesis is just not true? What more could you want? This is just a type of linear regression (it would be fun to actually have the P-value and correlation coefficient but it would be a very high correlation coefficient/small p-value). I get that they will wave their hands and say that transaction costs will eat up any profits but that does not save their hypothesis.


All,

When testing single factor ranking systems, if you want to limit or trim the ends, then instead of putting the formula FRank(“avgvol(10)/avgvol(120)”) < 99.5 in the universe, it is better to put it in the ranking system as a Boolean function. That way it is easier to include the single factor and the Boolean trim into a larger ranking system with many factors, and the universe won’t be messed up with a lot of trim functions for every single factor you need to trim.

Instead of two nodes, or a node and a universe rule, a single node modelled after:

avgvol(10)/avgvol(120) * (FRANK(“avgvol(10)/avgvol(120)”) < 99.5)

should accomplish effectively the same thing.

All,
thanks for the corrections and suggestions! It’s great to get so many solutions in a single thread :slight_smile:

Eval(Frank(“Pr2CashFlQ”) < 99.5, Pr2CashFlQ, NA) also works.

How often are you rebalancing when using volume?

The first thing to consider is the factor itself. Your second bullet point is not quite accurate, but it’s pointing in the right direction. Certain factors, especially growth rates and valuation ratios, are especially vulnerable to oddities that mean you’re not really seeing what you think you’re seeing. If a company’s TTM results are up a few hundred percent, is it because the company is really growing fast, or because it made a big acquisition? If a promising company has a P/E of 5, does that mean the market is missing out on a spectacularly undervalued opportunity, or were TTM earnings artificially high because they included a big gain book on the sale of an asset? Etc., etc., etc.

Use of a single-factor-model is inherently a horrifyingly bad idea for various reasons one of which is that it exposes you to this sort of problem. You need to use multiple ways of articulating the same idea (to hell with autocorrelation nonsense – that’s for statistics classes, in the real world, you need to diversify to mitigate unintentional specification error). The other is to really understand the data and work hard to create your own ratios in such a way as to guard against whacky things. The latter requires an understanding of fundamental business analysis and financial statements. The former is likely more palatable to quants who often aren’t knowledgable or interested in deep dives required by the latter.