Index | Recent Threads | Who's Online | Search

Posts: 99    Pages: 10    Prev 1 2 3 4 5 6 7 8 9 10 Next
Last Post
New Thread
This topic has been viewed 5834 times and has 98 replies
Jrinne
Re: Renaissance Technologies

Georg,

Thank you for the spreadsheet. And it is good to read that you agree with the premise of my post. I am relieved to find that I am not the only one who may be in need of shock therapy;-)

There seems to be an error with regard to the excess returns of you Best(SPY-SH) Gains for Up & Down Markets model, however.

You report excess returns of 39.38 see image.

I show the equity curve for this model illustrating this can be correct (second image).

To avoid accidentally making a similar error I just looked at the average 2 year excess returns of all of your models that had more than 2 years of excess returns for you. This is downloaded by P123 in a spreadsheet and does not require any calculation on my part (other than the spreadsheet finding the average of the column).

This was -6.24

People need to do this for all of the Designers that have an opinions in the Forum to see if what they post corresponds to their objective results. Sometimes it will (if you forget about survivorship bias).

Honestly, a full financial interest disclosure should have the Designer’s results as a part of every Designer’s post (without survivorship bias).

When will we stop chasing cherry-picked examples?

-Jim

Attachment Spreadsheet.png (38413 bytes) (Download count: 108)


Attachment Equtity curve.png (129853 bytes) (Download count: 106)


From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Dec 10, 2019 5:35:58 AM       
Edit 10 times, last edit by Jrinne at Dec 10, 2019 6:46:08 AM
geov
Re: Renaissance Technologies

Jim,
I picked off the 5-yr returns from the performance graph.
Here is what the return looks like today, with Monday's Dec-9 end date.

I am not saying that this return is particularly good, but at least it is better than buy&hold SPY. I will revise this model in line with my most advanced market timing strategy for better returns in the future.

Attachment SPY-SH 5Y=excess.png (57084 bytes) (Download count: 100)


Dec 10, 2019 7:44:23 AM       
Edit 1 times, last edit by geov at Dec 10, 2019 8:19:25 AM
Jrinne
Re: Renaissance Technologies

Geog,

Thank you for the clarification.

And clearly your models do better than most.

I still stress the importance of avoiding cherry-picked examples to people.

More broadly, you have confirmed that the math regarding the performance of all of the Designer models is correct. Thank you for alleviating that concern!!! Although I kind of wish we were both wrong on that point.

Very much appreciated. I do not like getting my facts wrong (which I did).

-Jim

From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Dec 10, 2019 8:16:21 AM       
Edit 5 times, last edit by Jrinne at Dec 10, 2019 8:22:55 AM
RTNL
Re: Renaissance Technologies

Me, if I were trading the black-box strategies (equal weight) from some of the Designers I would need more than a coach. Maybe some medication would help. But a coach would not cut it.
-Jim

One has to agree with Jim.
I made the exercise to check the 5-year returns of the Designer Models.
There are 75 DMs with inception 5 years ago or earlier.

The average total return over 5 years of the 75 models= 23.4%.
and the excess 5-year return over SPY= -43.8%

There are only 6 models which outperformed SPY over 5 years.
For the 6 models the average excess 5-year return over SPY= 25.8%

That is not great. We have to do better than that.
Perhaps good market timing is the answer.

Excel file is attached.


Thanks Georg for taking the time to do the study.
These are pretty shocking numbers. especially given the high turnover of many models. In search of outperformance, are we model builders guilty of over complicating things and building in sample optimized models/ ranking systems?

Dec 10, 2019 8:38:50 AM       
geov
Re: Renaissance Technologies

Jim, thank you for raising this topic in the first place.

I will now revise the Designer Model (SPY-SH) to include my most recent market timing rules. It should now perform better.

See attached simulated performance curve and stats.
Noteworthy is the correlation with SPY = 0.01.

Attachment SPY-SH revised model.png (161039 bytes) (Download count: 94)


Dec 10, 2019 8:58:27 AM       
Edit 1 times, last edit by geov at Dec 10, 2019 8:59:23 AM
Jrinne
Re: Renaissance Technologies

So the t-score on this is 11. This is for a single-sample t-test.

p-value? I cannot easily find any programs that will print out anything that small even using exponential notation. I may look later and amend this if I do.

Chances that Designer models are inferior to their benchmark over the most recent 5 year period? About as certain as you can be of anything in this life. Even with any survivorship bias that may exist--which would make it worse still.

-Jim

From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Dec 10, 2019 8:58:57 AM       
Edit 11 times, last edit by Jrinne at Dec 10, 2019 9:23:54 AM
geov
Re: Renaissance Technologies

Thanks Georg for taking the time to do the study.
These are pretty shocking numbers. especially given the high turnover of many models. In search of outperformance, are we model builders guilty of over complicating things and building in sample optimized models/ ranking systems?

I think designers want to show huge backtest returns. That is not a good strategy because those returns are not achievable going forward.

Designers should aim for a 20% annualized return, then everybody will be happy with 15% in real live. Nobody would mind getting a steady 15% on their money.

Dec 10, 2019 9:18:09 AM       
Jrinne
Re: Renaissance Technologies

Thanks Georg for taking the time to do the study.
These are pretty shocking numbers. especially given the high turnover of many models. In search of outperformance, are we model builders guilty of over complicating things and building in sample optimized models/ ranking systems?

I think designers want to show huge backtest returns. That is not a good strategy because those returns are not achievable going forward.

Designers should aim for a 20% annualized return, then everybody will be happy with 15% in real live. Nobody would mind getting a steady 15% on their money.


Georg,

I could not agree more!!!!

Honestly, I am wondering if Ray Dalio's 12% might be a more realistic goal for me personally when the dust settles. Anyway, whatever number I finally arrive at the number will be changed forever after this.

-Jim

From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Dec 10, 2019 9:26:07 AM       
Edit 2 times, last edit by Jrinne at Dec 10, 2019 9:27:58 AM
WERNER
Re: Renaissance Technologies

Interesting discussion.
Goes right into the critical issue: Do we have any reasonable conviction or even facts, to think that forward returns are in any way similar to the backtested ones?
Do we have any metric(s) or parameters which will show us overoptimization? That should accompany any model.

As Georg is rightfully saying:
"I think designers want to show huge backtest returns. That is not a good strategy because those returns are not achievable going forward."

Exactly.

Werner

Dec 10, 2019 9:54:09 AM       
Edit 2 times, last edit by WERNER at Dec 10, 2019 10:17:23 AM
yuvaltaylor
Re: Renaissance Technologies

Interesting discussion.
Goes right into the critical issue: Do we have any reasonable conviction or even facts, to think that forward returns are in any way similar to the backtested ones?
Do we have any metric(s) or parameters which will show us overoptimization? That should accompany any model.

As Georg is rightfully saying:
"I think designers want to show huge backtest returns. That is not a good strategy because those returns are not achievable going forward."

Exactly.

Werner


The answer to both questions is no.

We can assume that things that have worked in the past have a higher *probability* of working in the future than things that have totally failed in the past. In other words, stocks with very high accruals and very high price-to-sales ratios are likely going to keep underperforming stocks with low accruals and low price-to-sales ratios.

However, we can also assume that there will always be a fair amount of mean reversion going on as well. And we can assume that many widely used factors and/or approaches may end up being arbitraged away. And there can be massive "regime changes," black swan events, that will put everything out of joint.

As to how all those things will cancel each other out, we have no idea.

As far as I have been able to tell, there are no metrics that measure over-optimization with any degree of certainty. But there are questions that should be asked of any designer.

1. How does your system perform if you increase the number of holdings by four? by ten?

2. Have you tested your system on out-of-sample periods or universes, or on partial universes and/or partial time periods? And are the results consistent?

3. Have you built your system by constantly tweaking minor details to get a better and better backtested performance? If so, it may be best to stop doing that--it may be detrimental to out-of-sample performance.

I have built six designer models. Three of them have significantly outperformed their benchmarks. One of them I removed because I had lost confidence in it. The other two are underperforming their benchmarks, but only barely. I have been guilty of over-optimizing as well--I built all six models by constantly tweaking minor details in order to get optimal backtested performance. I don't know whether that proves anything. The oldest of these models have been live since early 2017--over two years now. I also have been using optimized models for live trading since late 2015, during which time I have maintained a CAGR of 30%. My live trading has underperformed in the past year or so, but it has still been profitable (13% YTD), and because I aim for low beta, I expect to underperform when the market is going gangbusters. So in my PERSONAL experience, over-optimizing has not been harmful. I still think, however, that it's not a good thing to do.

Yuval Taylor
Product Manager, Portfolio123
invest(igations)
Any opinions or recommendations in this message are not opinions or recommendations of Portfolio123 Securities LLC.

Dec 10, 2019 11:29:10 AM       
Posts: 99    Pages: 10    Prev 1 2 3 4 5 6 7 8 9 10 Next
 Last Post