P123 ranking systems underperform the market?

[Monday33]

Hi all.

I’ve been working a little with the ranking systems P123 have in their system, the 34 of them, and I am a little bit concern about some facts I would like to comment with you.

First of all I created one sim, and then I just changed the ranking system and see what happeds in different periods of time, and in different rebalancing periods. The idea behind my “movement” was to see which ranking performs better.

The surprise where that while almost ALL of them work pretty well before 2008, ALL of them underperfom the market from 2008 and ahead.

The benchmark (market) in my sim, is the S&P500, and for my first test I pick the stocks (Universe) in all the US stocks except in the OTC market.
Only one buy rule;

Price > 1 & AvgVol(20) > 50000

And one sell rule;

rank <90.

That’s it.

First I did it with 100 stocks, then with 20.

Then, I keep the benchmark, but I pick the stocks from the S&P500 itself, just to know if the rankings of P123 have some edge over the benchmark.

The answer was NO (talking about 2008 and ahead). And it’s very surprising to me. Almost all of the P123 rankings underperform the S&P500, or went clearly under the benchmark.

It looks like the market conditions changed a lot, to 2008 from now, and we are just digging with the wrong tool, a one that used to work pretty well, but nowadays don’t.

And, of course, if I add some extra buy rules, to the sim, the performance improve, but I’m little bit confuse about it and I do not know what to think about.

Did you experience the same?

What happened here? Any ideas?

It’s time for P123 to create new ranking systems?

[yuvaltaylor]

This is an EXCELLENT observation, and I want to thank you for making it.

It has become very difficult to beat the S&P 500 using stocks from the S&P 500 and conventional metrics. A few of our ranking systems do it, but very few (Greenblatt, Basic Value, Yield-Growth Combination, Foolish 8, Value Sentimentum), and even those don’t beat it by much (I used a screen to backtest the top 50 stocks in the S&P 500 over the last ten years, rebalancing monthly). The S&P 500 is a collection of very large and very high-quality companies. Investing in those companies is a good idea. Choosing a few of them and trying to beat the others may not be.

That’s why most successful users of Portfolio123 avoid large caps and concentrate on small caps and microcaps. The possibility of outperformance is much higher, because that’s where the gems can be found.

A few points I’d like to make along these lines.

First, besides (or instead of) using simulations to test this out, you can easily do so with a screen, or, even better, look at the ten-bucket performance of the ranking system over the last ten years and the S&P 500 universe (see screen shots below). This is the best way to evaluate ranking systems.

Second, in order to have an edge over the market, you need to use unconventional but effective metrics. There are plenty of value, quality, and growth ratios that one can use that aren’t used in the ranking systems because those depend on older, more conventional metrics. You may not find these in the wizard; you may have to glean them from reading financial literature. One good idea might be to read Seeking Alpha articles about individual stocks and look at the metrics the writers are using to analyze them. Another might be to concentrate on stocks in specific industries or sectors and develop metrics tailored for those. Very few of the metrics that I use are in the old ranking systems that were mainly developed prior to the GFC.

Third, I am willing to create some new P123 ranking systems if there’s a real demand for them. Let me know what you’d like to see in a P123 ranking system, and I’ll talk about it with the rest of the folks here.

---

---

[InmanRoshi]

I was listening to a finance podcast the other day, and the hosts were mentioning that that they personally knew someone who recently had dinner with Charlie Munger. The question was posed to Munger how he would have approached running Berkshire if he were just starting it up today. Munger said the main thing he would do is never invest in companies that are in the major indexes or ETFs.

[primus]

There are plenty of value, quality, and growth ratios that one can use that aren’t used in the ranking systems because those depend on older, more conventional metrics. You may not find these in the wizard; you may have to glean them from reading financial literature. One good idea might be to read Seeking Alpha articles about individual stocks and look at the metrics the writers are using to analyze them. Another might be to concentrate on stocks in specific industries or sectors and develop metrics tailored for those.

Excellent advice!

[Jrinne]

Yuval has great points.

On, what I think is a different (but related) subject, I have been wondering about whether I should be updating or re-optimizing my ranking systems.

I have been thinking about what we do in the context of machine learning for a while.

I now think of the ranking system in this way. After I developed my ranking systems I ran them live and found some worked. For the ones that worked I validated the METHOD—including the factors I used.

Here, 6 years later, if I were to re-optimized the weights on the factors—that would be consistent with most machine learning methods of cross-validation. It would be like finding that a linear regression works (using an out-of-sample test set) and running the regression again with all of the data—including the test set. Then using the regression (in this example) with all of the data going forward

The constants will change when the regression is run on all of the data. But the METHOD has been validated. Adding the new data (from the test set) is only giving a validated method more data to work with—which should make the model even better.

So, maybe some of the old ranking systems should be discarded based on the out-of-sample performance. For the ones that are working, re-optimizing WOULD NOT CONFLICT WITH THE BEST PRACTICES DESCRIBED IN MOST TEXT BOOKS.

I will not be adding any new factors as that could lead to overfitting. But I may get rid of some factors. I plan to re-optimize by getting rid (or reducing the weights) of some factors that no longer work and updating the weights on the factors that do work.

-Jim

[mgerstein]

The benchmark (market) in my sim, is the S&P500, and for my first test I pick the stocks (Universe) in all the US stocks except in the OTC market.
Only one buy rule;

Price > 1 & AvgVol(20) > 50000

And one sell rule;

rank <90.

That’s it.

Ouch. I’d never invest that way. Depriving yourself of the benefits of screening/buy rules is like a sales person prospecting from the phone book instead of a lit of pre-qualified prospects.

As to the p123-user preference for small and micro caps, that’s fine but make sure you understand WHY you got the results you did. It’s mostly because you were risk-on (full spigot) at a time when risk-on was heavily rewarded.

Understand the role the Fed has been playing in pumping this part of the market. When the cost of capital is pushed way, way, way down, it brings in the most junky least-worthy buyers. We’ve had a ton of that for most of the time since 1982. But watch out for how those p123 favorites have done during times when the market started leaning in a different direction. It didn’t last yet. Rates are still bottom feeding. But a lot of first-generation designer models imploded because without realizing it, a lot of those models counted on great times continuing, and even the teeny weeny bit of non-looseniing we experienced after DM debuted was enough to blow up a lot of models.

If you want to stay small or micro, that’s fine. But don’t be complacent that you’ll keep seeing what you’ve seen. Stay alert for changing conditions.It’s easy to become a prisoner of the moment when the moment has lasted the better part of 35 years.

Do yourselves a favor. Teach yourself to strategize among mid and large cap stocks. Even if you keep them as paper models for a while, do yourself a favor and learn how to build them, so you’ll be ready for when you need them. You’re lucky. The market is giving you time to learn and make learner mistakes along the way. Use the time wisely. Don’t squander this period by assuming you’ll always want to rush to the bottom.

[geov]

It has become very difficult to beat the S&P 500 using stocks from the S&P 500 and conventional metrics. A few of our ranking systems do it, but very few (Greenblatt, Basic Value, Yield-Growth Combination, Foolish 8, Value Sentimentum), and even those don’t beat it by much (I used a screen to backtest the top 50 stocks in the S&P 500 over the last ten years, rebalancing monthly). The S&P 500 is a collection of very large and very high-quality companies. Investing in those companies is a good idea. Choosing a few of them and trying to beat the others may not be.

That’s why most successful users of Portfolio123 avoid large caps and concentrate on small caps and microcaps.

I don’t agree with Yuval that large-caps should be avoided in preference to small caps. Problem is that most people have an unrealistic expectation of beating the market by a lot and think that rapid trading strategies will do it.

Here is the performance of a model that picks 10 stocks from the +/- 42 of the Vanguard VDIGX Growth and Income fund, all of them large-caps. (Why bother selecting your own universe when more experienced people have already done it for you.) If the ranking system was useless then one would expect to do no better than the fund; after all my model holds a quarter of the fund’s stocks. But the ranking system works fine, the Best10(VDIGX)-Trader is now live for nearly 5 years and shows a total return almost double that of SPY or VDIGX, with low turnover of 167%.

Today’s report: The model’s out of sample performance YTD is 12.0%, and for the last 12 months is 14.7%. Over the same period the benchmark SPY performance was 12.9% and 4.9% respectively. Since inception, on 7/1/2014, the model gained 105.45% while the benchmark SPY gained 57.61% and VDIGX gained 61.58% over the same period. Over the previous week the market value of iM-Best10(VDIGX) gained -1.96% at a time when SPY gained -4.08%. A starting capital of $100,000 at inception on 7/1/2014 would have grown to $205,453 which includes $9 cash and excludes $2,511 spent on fees and slippage.

This model has since inception an annualized return of 15.9%, 6% higher than the 9.8% for SPY. Any large-cap model that consistently beats SPY by 5% is exceptional.

---

---

[Monday33]

You wellcome, I tried to do my best. But thanks to you for your answer.

Third, I am willing to create some new P123 ranking systems if there’s a real demand for them. Let me know what you’d like to see in a P123 ranking system, and I’ll talk about it with the rest of the folks here.

About that, I think what I like to see in P123 ranking Systems is ranking Systems that work in our contemporary environament, not 10 or 20 years ago. If not, there is always the temptation to go and buy the index and spend our time whatching TV instead of doing some reserch here.

And maybe a more specific ranking systems, like oriented to a specific industries, sectors, or subsectors, can be extreamly usefull here. A lot more than generic ranking systems that try to work for all-and-none.

[Monday33]

Ouch. I’d never invest that way. Depriving yourself of the benefits of screening/buy rules is like a sales person prospecting from the phone book instead of a lit of pre-qualified prospects…

I do not either. It’s just a way to compare different elements between them.

[Monday33]

This model has since inception an annualized return of 15.9%, 6% higher than the 9.8% for SPY. Any large-cap model that consistently beats SPY by 5% is exceptional.

But, did you add some buy and sell rules to the model, or it just Works with the ranking, first ten positions, and nothing else?

[InspectorSector]

Monday - when you first arrived here, I had you perform an exercise involving the creation of a largecap sim using the Greenblatt ranking system. (In my opinion, the results were great.) But you seemed to have jumped around from there. What happened to the sim? If you still have it then how would it have performed since you developed it?

[geov]

The Best10(VDIGX)-Trader has no buy rules. You don’t need any because you are already working with a pre-selected universe of growth and income companies. There is only one rank based sell rule.

You can’t do any reliable backtesting with the current VDIGX universe because the simulated performance will be much higher due to survivorship bias, than what it would have been live.

[Monday33]

Hello Steve.

I could not improve the sim more since the last time we talk about it, and I was more or less happy with the results, then wrote and article about it in my blog, and did not work more on it.

A short history.

Anyway I do not trust so much in the “Magic Formula” since the latest poor results it is having.

[WalterW]

mgerstein: Ouch. I’d never invest that way. Depriving yourself of the benefits of screening/buy rules is like a sales person prospecting from the phone book instead of a lit of pre-qualified prospects.

Do yourselves a favor. Teach yourself to strategize among mid and large cap stocks. Even if you keep them as paper models for a while, do yourself a favor and learn how to build them, so you’ll be ready for when you need them. You’re lucky. The market is giving you time to learn and make learner mistakes along the way. Use the time wisely. Don’t squander this period by assuming you’ll always want to rush to the bottom.

Perhaps I have an idiosyncratic style, but performance oriented buys-rules frequently, if not always, hurt my out-of-sample performance. In a recent case, technical buys rules improved the equity curve, but a quick rolling test showed that the excess alpha occurred in the pre-2008 years. Currently, the only buy-rules I consider are to filter sectors/industries.

For me, the ranker does the heavy lifting. I’m lucky to live in an area that has a lot of Meetups. At a recent meeting, I was chatting with a data scientist who works for a large investment group. After discussing my DM S&P dividend model, he casually mentioned that I probably found that I needed only four or five factors to make it work. And he was right! The ranking component is very powerful. Don’t neglect it.

However, I can enthusiastically endorse basing a model (i.e. ranker) on a strategy. My first dividend model targets certain types of dividend paying companies. That approach seems to work - again, for me.

Monday33:
About that, I think what I like to see in P123 ranking Systems is ranking Systems that work in our contemporary environament, not 10 or 20 years ago. If not, there is always the temptation to go and buy the index and spend our time whatching TV instead of doing some reserch here.

Anyway I do not trust so much in the “Magic Formula” since the latest poor results it is having.

I like to tell people that P123 is a site for the thinking investor. You will really need to put in the time and read, and read, and read and sim, and sim, and sim, to get the full benefit. Being given a system does nothing but create investing monocultures.

BTW, I think the Greenblatt Magic Formula ranker is great given its simplicity.

Walter

[InspectorSector]

Anyway I do not trust so much in the “Magic Formula” since the latest poor results it is having.

Monday - What are you basing your “poor results” on. Can you show me the performance of that sim or port since it was designed? The Greenblatt ranking system has been around since ~2013 I think and is still working for me.

[Monday33]

When I’m talking about “poor results” I only mean the results that guy* achive by following the Greenblatt System. He does this from the page Greenblatt sets online, that gives a “buy list” of stocks, and the guy does this with real money.

*http://www.magicformulaexperience.com/

Are you using that System since 2013, with real Money?

Then if it Works for you maybe the buy list the web page offers it’s not working pretty well.

[RTNL]

Large caps work. I think you have to buy the stock as if you would buy the business. Here is an example. Base universe is R1000

---

[InspectorSector]

Monday - I can’t speak for someone on a different site with a different design. What I am asking is how the portfolio design that you created based on my instructions has performed since you finished with it. If you still have the sim or port then we can look at recent results and discuss whether or not it is “still working”.

Steve

[Monday33]

Monday - I can’t speak for someone on a different site with a different design. What I am asking is how the portfolio design that you created based on my instructions has performed since you finished with it. If you still have the sim or port then we can look at recent results and discuss whether or not it is “still working”.

Steve

Of course Steve, you right. But if you don’t mind I’m going to post the results of that simulation in the thread I created to talk about that sim, because I think it’s better not to mix themes, I opened this thread only to talk about the underperformance of the P123 ranking systems from 2008 onwards.

By the way, I can say that if I start that sim from 2008 onwards it’s underperforming the market.

[InspectorSector]

So let us stay on this thread for a while if you don’t mind. This thread is about P123 ranking systems underperforming the market. Yet when I queried you, you pointed to some other website and someone else’s implementation of Greenblatt, not P123s ranking system.

The model that I was helping you with was based on the S&P 500 stock universe. Your other thread last post “results” is for a Non-OTC Universe, My first observation is that you are jumping around way too much to come to any conclusions. The Non-OTC universe is very volatile and of course you will get high beta which is what I am seeing in your results.

What you need to do is get back to the post in the other thread, the post with results based on the S&P 500

“I just mixed your instructions with the ones from Yuvaltaylor and because of that I place the s&p1500 universe, but that was a mistake. Just I would like to ask you why do you prefer the RSP as a benchmark than the SPY? The only difference I can see between them it’s that the RSP offers more performance on the long term.”

If you can find that sim, then start there and let us have a look at it’s out of sample performance.