Index | Recent Threads | Who's Online | Search

Posts: 99    Pages: 10    Prev 1 2 3 4 5 6 7 8 9 10 Next
Last Post
New Thread
This topic has been viewed 4842 times and has 98 replies
judgetrade
Re: Renaissance Technologies

keating ranking, 3 Month rebalanced not changed since then...

Attachment aaa.png (346979 bytes) (Download count: 79)


Dec 12, 2019 7:05:15 AM       
judgetrade
Re: Renaissance Technologies

used the same ranking from 2011 - 14 too... not optimized... was able to beat transaction costs realtime too!

Dec 12, 2019 7:07:10 AM       
InmanRoshi
Re: Renaissance Technologies

Yes. It appears all quants are "over optimized", even the old time pros like Assness and O'Shaughnessy, if they've been buying value or quality or small caps the last few years.

Dec 12, 2019 8:06:21 AM       
Edit 2 times, last edit by InmanRoshi at Dec 12, 2019 8:26:50 AM
Jrinne
Re: Renaissance Technologies

Andreas,

Make sure to share some of your abilities in the Designer Models when you get a chance. You have enough models that your average result on these models would be good if you had a secret that you are sharing with us in these models.

I am happy to keep the discussion to Olikea or the all of the Designers as a group if people would not make anecdotal claims that cannot be verified. This is important enough to P123 that we need objective data, I think.

Anecdotal stuff, Cherry-picking needs to stop if P123 is to survive.

Fundamental analysis until we buy black boxes that do not work is a business model that can be expanded upon, IMHO. Andreas’ model aside: it is not a black box perhaps. Maybe we can just all use his model. I might even subscribe to help Andreas. I will want out-if-sample result before I put any money in it.

-Jim

From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Dec 12, 2019 9:11:57 AM       
Edit 3 times, last edit by Jrinne at Dec 12, 2019 10:21:22 AM
Jrinne
Re: Renaissance Technologies

Yes. It appears all quants are "over optimized", even the old time pros like Assness and O'Shaughnessy, if they've been buying value or quality or small caps the last few years.


Good to know.

With regard to P123 we manage to underperform even value benchmarks when used.

One could repeat Georg’s study where a value index is used as benchmark for all models to get more information on this hypothesis. Not sure what it would show.

Georg presented convincing evidence for 5 years. Not a few.

But you present a hypothesis that will be tested with time.

I only suggest that we do a little active testing. If people would prefer to just let Designer Models run for another 5 years and see what comments we get in the forum it is a plan at least.

BTW, has anyone seen a p-value like Georg gets? I understand we are to accept anything we are linked to over at SeekingAlpha but not the best p-value you have ever seen.

Good luck with that.

-Jim

From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Dec 12, 2019 9:27:41 AM       
Edit 4 times, last edit by Jrinne at Dec 12, 2019 10:34:32 AM
geov
Re: Renaissance Technologies

So here is the evidence that backtesting does not provide a good indication for OOS performance:

I have selected 25 DMs with large + mid caps >=70% all with inception date earlier than 5 years ago and put them into a book. (25 models is the max allowed in a book.)

Look at the backtest from 2002 to 2014. That is the backtest period which designers considered. From Figure-1 one can see that designers did very well. Annualized return= 28.6% with a max D/D= -18%. The 2008 financial crises is not even visible on the performance curve. Calendar year performance is equally impressive - every year has positive returns all exceeding that of SPY, all as shown in Figure-2.

So why did this great simulated performance not continue over the 5-year out-of-sample period 12/1/2014 to 12/2/2019 (Figure-3)?

Almost immediately the combo starts under-performing SPY, over 2015 by -4.0%. How can that be when for each of the preceding 13 years it out-performed SPY?

Over the last 5 years the annualized return= 5.0% with a max D/D= -21%. Calendar year performance is equally unimpressive - every year the 25 DMs underperformed SPY, 2015 to 2019: -4.04%, -6.34%, -4.33%, -4.89%, -11.48%.

Performance relative to Value is not much better. Calendar year performance relative to IWD is equally unimpressive - 2015 to 2019: 1.17%, -11.60%, 3.93%, -1.01%, -7.20%.

Attachment Fig1 25 Large-Mid Cap DMs from 1-1-2002 to 12-1-2014.png (145752 bytes) (Download count: 60)


Attachment Fig2 25 Large-Mid Cap DMs Yearly from 1-1-2002 to 12-1-2014.png (33458 bytes) (Download count: 64)


Attachment Fig3 25 Large-Mid Cap DMs OOS from 12-1-2014 to 12-2-2019.png (181799 bytes) (Download count: 60)


Dec 12, 2019 1:12:08 PM       
Edit 5 times, last edit by geov at Dec 12, 2019 5:18:35 PM
Jrinne
Re: Renaissance Technologies

So here comes the proof that backtesting does not provide a good indication for OOS performance:


Georg,

Thank you.

Shouldn’t P123 hope this is overfitting? I think they should.

Twice you mentioned AIC which will stop overfitting in its tracks. I leave it to you to try to explain how you would implement this to the staff at P123 and the members—if you think anyone (other than me) will listen to you the third time. I think they should.

AIC is good. I also like LASSO regression. That would mean using that m x n array I keep mentioning to do a LASSO regression. But once the features are selected you could move back to a rank performance optimazation with the selected features. BTW, LASSO or AIC would not use much computer resources.

Result: no more overfitting. Gone, nada, none. Zero, zip overfitting. The big goose egg….

Ultimately I would use something related, myself, but I doubt we will get past AIC or LASSO regression in any discussion.

Anyway, I think P123 should hope the problem is the easily addressed problem of overfitting and give us the tools to end the problem (if that is what the problem is). P123 should hire a consultant if Georg is not understood by P123 staff this third time.

It is NOT rocket science. But you cannot read a post about this somewhere and become an expert either. Georg has obviously studied this to know this as well as he does. But I am going to stick my neck out and say that despite his obvious training and credentials he might not have been a "Rocket Scientist."

Okay, it helps to be a Rocket Scientist or Engineer like Georg. People should really listen to his ideas about AIC and maybe some of the other methods of addressing overfitting. If overfitting is the cause of the poor Designer Model performance if would probably pay to hire a consultant to make sure Georg is understood.

-Jim

From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Dec 12, 2019 2:45:00 PM       
Edit 25 times, last edit by Jrinne at Dec 12, 2019 4:09:18 PM
o806
Re: Renaissance Technologies


I would hazard the following guesses:

1. The old, tried-and-true factors like ROE and P/E have largely been arbitraged away, and stocks are priced in such a way that there are very few values to be found by using those factors. Investors are now so used to pricing stocks by using these metrics that it's hard to find any advantage in using them.

2. We may be in a regime that resembles the late 1990s, when overpriced high-growth tech stocks ruled the market and S&P slaughtered all other strategies in sight. As we all know, that was not meant to last.

3. While there certainly have been some egregiously over-optimized designer models, given the "base rate" I have established, we cannot blame over-optimization entirely for their overall failure to perform. Some of the "failed" models may actually be very good ones that are just going through a blue streak. After all, every single tried-and-true strategy and every single great investor has had some long periods of underperformance.

Yuval:

In my opinion your first two points are very plausible, and your final one is an absolute certainly.

From a logical and behavioral perspective point 3 has to be true --- if a model or group of related models did not have "a blue streak ... some long periods of underperformance" then everybody along with all their siblings and cousins would eventually be using the model and it would then stop working for --- and this is key --- stop working for long enough to get the majority of people to stop using it. At that point it would have a good probability of starting to work again.

Put in other words, if a method is going to work over the long term it has to inflict enough pain, from time to time, along the way to get most people to quit. It is a variation of "no pain, no gain".

Our data only goes back to 1999, but I've heard some who have access to data from the 1990s say that value and small caps did not work very well compared to the growth and larger caps during that period. From that perspective it is not surprising that the typical P123 portfolio (mine included) back tested very well from 2000-2002 (general bear market for the large indexes and range bound for the R2000) and continued to do outstanding well for 2003-2007. Many of our models did well for 2009-2017. Our methods are currently in a 1.5 year period of painful underperformance.

Will this period of underperformance be over in 3 months or 3 more years. I have no idea of the duration but I believer it will need to be long enough and painful enough to get the majority of money to stop using the types of value and small cap models that worked will in the past.

Well that's my 2 cents.

Brian

Dec 12, 2019 4:41:29 PM       
Chipper6
Re: Renaissance Technologies

You've probably heard that a picture is worth a thousand words. Here is one of those pictures. In order to make the right decisions, you need to know the facts.

As you can see, this older ranking system did its job even over the most recent twelve tough months. Yet, it did not beat SPY!

Think: How could SPY beat all the buckets? What does this tell us about what happened? What does this tell us about the power of the ranking systems?

Suggestion: Use the correct benchmark. Russell 3000 equal weight is close enough, but an equal weight of the sims universe is even better, especially for targeted universes.

A DM might be working. Or it might not. But what do you learn by comparing it to the SPY?

Attachment Annotation 2019-12-12 175330.png (27516 bytes) (Download count: 45)


Dec 12, 2019 5:18:58 PM       
Jrinne
Re: Renaissance Technologies


I would hazard the following guesses:

1. The old, tried-and-true factors like ROE and P/E have largely been arbitraged away, and stocks are priced in such a way that there are very few values to be found by using those factors. Investors are now so used to pricing stocks by using these metrics that it's hard to find any advantage in using them.

2. We may be in a regime that resembles the late 1990s, when overpriced high-growth tech stocks ruled the market and S&P slaughtered all other strategies in sight. As we all know, that was not meant to last.

3. While there certainly have been some egregiously over-optimized designer models, given the "base rate" I have established, we cannot blame over-optimization entirely for their overall failure to perform. Some of the "failed" models may actually be very good ones that are just going through a blue streak. After all, every single tried-and-true strategy and every single great investor has had some long periods of underperformance.

Yuval:

In my opinion your first two points are very plausible, and your final one is an absolute certainly.

From a logical and behavioral perspective point 3 has to be true --- if a model or group of related models did not have "a blue streak ... some long periods of underperformance" then everybody along with all their siblings and cousins would eventually be using the model and it would then stop working for --- and this is key --- stop working for long enough to get the majority of people to stop using it. At that point it would have a good probability of starting to work again.

Put in other words, if a method is going to work over the long term it has to inflict enough pain, from time to time, along the way to get most people to quit. It is a variation of "no pain, no gain".

Our data only goes back to 1999, but I've heard some who have access to data from the 1990s say that value and small caps did not work very well compared to the growth and larger caps during that period. From that perspective it is not surprising that the typical P123 portfolio (mine included) back tested very well from 2000-2002 (general bear market for the large indexes and range bound for the R2000) and continued to do outstanding well for 2003-2007. Many of our models did well for 2009-2017. Our methods are currently in a 1.5 year period of painful underperformance.

Will this period of underperformance be over in 3 months or 3 more years. I have no idea of the duration but I believer it will need to be long enough and painful enough to get the majority of money to stop using the types of value and small cap models that worked will in the past.

Well that's my 2 cents.

Brian


Good thoughts. And ultimately it may take a lot of ideas to express our present situation.

The idea that there are a series of events for us to have only 8% of Designer Models beating their benchmark over a 5 year period is a good one. And as Georg said:

the excess 5-year return over SPY= -43.8%


Not wrong to think there may be multiple problems including the above ideas, I would guess. I am not sure it is even possible for just one thing to do this.

I think posts implying there is just one problem--so no worries--are pure speculation.

BTW, Does equal weighting remove the problem of overfitting? I think not. Otherwise, feature selection to remove noise factors would not be the standard of for statistical learning. Look to Georg’s suggestion of AIC for an informed idea about removing noise factor that cause overfitting. You may need to search it for now.

Generally, factors are removed and not given equal weight to prevent overfitting.

Which is not to say overfitting is the only problem. But I see no evidence that it is not one of the factors—not in this thread for sure. I do think P123 should hire a consultant to give us the tools to end overfitting when it is a problem for a Designer.

-Jim

From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Dec 12, 2019 5:23:11 PM       
Edit 13 times, last edit by Jrinne at Dec 12, 2019 5:56:56 PM
Posts: 99    Pages: 10    Prev 1 2 3 4 5 6 7 8 9 10 Next
 Last Post