Rule: Universe or Screening?

Research
1

I have the following rule:

EPSActual(0,QTR) > 0

I’m getting much different backtest results depending on whether this rule is implemented as part of the Universe or as a screening rule.

What is the proper place for this rule, Universe or Screen?

I have a feeling the backtest results I get when this rule is applied in the Universe, are illusory.

Thank you.

2

I just got the same result with a custom universe vs screening rule with the formula you gave.

Can you send us the name of the screen(s) or strategies in question ?

Thanks

3

I think I understand what you’re asking: you have an investment system consisting of various rules and a ranking system. You’re trying this particular rule in the universe and in the screen, and you are getting very different results because if it’s in the screen, the ranking system ranks all stocks whether or not they have positive EPS, and if it’s in the universe, it only ranks stocks with positive EPS.

My analogy is as follows. Putting a major exclusionary rule in the screen instead of the universe is like trying to decide who gets to play on the Nigerian national basketball team while including in your list players who live in Cameroon and Senegal. All those extra players are irrelevant and will make it harder to choose the right players. That’s why I think major exclusionary rules like this should always be put in the universe.

For a broader discussion of this, see How to Design a Fundamentals-Based Strategy That Really Works, Part Two: Applying Rules and Designing Ranking Systems - Portfolio123 Blog

4

Nonetheless, the EPS filter should filter out EPS <= 0 regardless of the rank, so shouldn’t the result of this ranking and filtering be the same in either case?

This leaves me with the dilemma of deciding which backtest result represents reality and which is an artifact.

I’ll see if I can set this up to demonstrate it.

5

If you go into my account, there is a screen called “Test Screen” and a custom universe called “Test Universe”.

The EPS > 0 filter appears in both the test screen and test universe. If you try enabling the filter in one and disabling it in the other, then try it again reversing the screen and the universe, you will get two different annualized returns.

6

I looked at your screen and tested it on various dates. In each case the same stocks passed the screen whether the rule was in the universe or in the screener.

If your backtests differ, that’s probably because at some point one or two stocks differed.

You ask how this can happen?

I’ve made an Excel spreadsheet that illustrates this well, if you know Excel.

In column A are a list of imaginary stocks. In columns B, C, and D are their ranks on three imaginary factors. In column H are their earnings: about 2/3 of them are positive. In column F are the sums of the 3 factors’ ranks, then excluding any stocks with negative earnings. In columns J, K, and L I put the three factors from columns B, C, and D, but excluded the stocks with negative earnings. So they rank slightly differently. In column N is the sums of the ranking of those fewer stocks. The third-highest-ranked stock is different in columns F and N, simply because each stock is being ranked against fewer competitors.

illustration of ranking with a rule before and after screening.xlsx (12.2 KB)

7

Thank you for looking into this, Yuval.

Unfortunately it does not solve my dilemma of where the EPS > 0 filter correctly belongs and which backtest is valid and hopefully predictive.

Here is a sample of the different annualized returns I get depending on where the EPS filter is:

Filter in universe only: 66.27%

Filter “ON” in both universe and screen: 66.27%

Filter “OFF” in both universe and screen: 74.86%

Filter in screen only: 50.36%

It goes without saying that the annualized returns are different because different stocks are being picked.

Four logical conditions and three different annualized returns. Which one is valid and which one do I trade going forward? That is my dilemma. Do I want to omit the EPS filter altogether?

The difference between 50.36% and 74.86% is too large to ignore.

8

For a general sense of what makes a backtest predictive, I suggest you read these two blog articles:

https://blog.portfolio123.com/how-to-design-a-fundamentals-based-strategy-that-really-works-part-three-principles-of-backtesting/

https://blog.portfolio123.com/break-your-strategy-how-to-stress-test-your-quantitative-models/

Like I said, it’s better to put your exclusionary rules in your universe, no matter what your backtest results are. That way you filter out a lot of noise.

You could also try to see how different the actual stocks picked are between your various backtests. If you download the data and compare each week’s returns, you’ll find the ones that are different and you can zero in on those. Most of the time, I think the same stocks will be picked.

I don’t know how many stocks you’re backtesting, nor what your liquidity limits are, but in general I’d advise you never to backtest fewer than 25 stocks and never to backtest without some realistic liquidity limits to eliminate penny stocks and stocks you’d have a very hard time trading. From your annualized returns in the above post, I would suspect you might be testing too few stocks or including stocks you’d be unable to trade. Also, make sure you’re taking slippage costs into account.

9

I can omit the EPS > 1 filter entirely without adversely affecting the overall system, so that’s what I’ll do on account of the ambiguity in the backtest results.

Price >= 1 filters out penny stocks and also makes a big difference in returns. This is no big deal because I can sacrifice some return in exchange for liquidity. I have already run into liquidity issues in my taxable account which takes much larger positions than my IRA.

I tested my system with 30 stocks and it still beats the benchmark (VTI) by a wide margin.

The system beats the benchmark with 3% slippage.

I have devised a way of analyzing the sequence of annual returns to give an idea of the odds of the system beating the benchmark. Unfortunately we only have past data to work with so it doesn’t speak to predictability.

10

This is true, but what you describe shouldn’t affect the final sort. For example:

No filter applied to universe:

A
B
C
D
E
F
G

The stocks are sorted, then the filter is applied in screening:

A
B
C
E
G

Stocks D and F have EPS <= 0

Now let’s apply the filter to the universe, then sort:

A
B
C
E
G

Stocks D and F were omitted from the universe. The final sorted list is exactly the same.

If the stock picks are identical then the annualized returns should be identical whether the EPS filter is in the screen or the universe. The fact that they are not identical calls into question the validity of the backtest.

Let’s keep in mind that EPS is a dynamic number and changes from quarter to quarter and we are using point-in-time data.

11

You’re omitting from your logic the fact that the sorts could be different with and without the excluded stocks because these are multifactor sorts. If every stock was being sorted on only one factor, then, yes, your logic applies. But please take a look at the Excel file I attached to my earlier post, which shows you an example of how sorting on three factors can produce different results depending on whether stocks are excluded prior to or after sorting.

I don’t know how else I can convince you of this, but if you still have doubts, let me know, and I’ll try some other way. I would like to convince you that the fact that a system will occasionally choose different stocks depending on whether stocks are excluded before or after multifactor sorting does not invalidate the procedure.

12

To test this claim I changed the ranking to a single-factor sort (PEG). Now it is no longer a multifactor sort.

Here is a sample of the different annualized returns I get depending on where the EPS filter is:

Filter in universe only: 24.60%

Filter “ON” in both universe and screen: 24.60%

Filter “OFF” in both universe and screen: 32.35%

Filter in screen only: 31.98%

Clearly it makes a difference whether the EPS filter is in the screen or the universe, even if it is not a multifactor sort but a single-factor sort.

13

Chris - I can’t see how that could happen. I tried it myself and the results are the same. Could you perhaps post the URLs of the particular screen, ranking system, and universe you’re using? Or at least their names? Thanks!

14

“Test Screen”

“Test Universe”

All you have to do is toggle the EPS > 0 rule on both screens, for a total of 4 backtests.

15

x

16

You have the rule RankPos <= 3 in your screen. So if you put your exclusionary rule in the universe, you’ll always get 3 stocks passing your screen. But if you put your exclusionary rule in the screen, you’ll sometimes get only 0, 1, or 2 stocks passing the screen, since the highest-ranked stocks might have EPS < 0.

I hope this explains the difference to your satisfaction. If you want to duplicate the results between the screen and the universe, then eliminate that RankPos <= 3 rule and instead use the Max # of stocks box in the Settings tab to choose the top stocks.

17

Yes, thank you for the explanation which is more than satisfactory. It restores my confidence in P123.

Interestingly, the highest annual return in obtained by omitting the EPS > 0 filter entirely.

Thanks again, Yuval.