Index | Recent Threads | Who's Online | Search

New Thread
This topic has been viewed 256 times and has 8 replies
Jrinne
I have some data (port vs sim)

Can others share their simulated live strategies to see if they're getting the same thing as me? Surely we all have live strategies going. Just simulate it, copying it exactly.

Nisser,

Assuming you do not think this is a closed issue and that you have not already seen all the data you need to see, I think I have some data to share.

I seem to have forgotten my auto ports. Sorry I did not think of them till now.

Here is the very first (non-cherry picked) port and sim. I have not looked at any others yet, I ran the sims with both preliminary data and exclude preliminary data. No revisions or discretionary behavior that I forgot about on these auto ports for sure.

The port is an auto port started years ago. It looks like it defaulted to Use Preliminary Data. The sims were set to Next Average of Hi and Low and Variable Slippage before running.

These are all over the same 3 month period since we switched to FactSet. Roughly a difference of 4% over a 3 month period. This was a 10 stock model.

More to follow.

Jim

Attachment Port.png (183306 bytes) (Download count: 91)


Attachment Sim Use Preliminary data.png (180845 bytes) (Download count: 92)


Attachment Sim Exclude Preliminary data.png (158650 bytes) (Download count: 90)


From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Oct 9, 2020 5:23:17 PM       
Edit 8 times, last edit by Jrinne at Oct 9, 2020 6:15:30 PM
Jrinne
Re: I have some data (port vs sim)

Nisser, Brett, Chaim and RT,

I looked at 30 ports that are on automatic. Almost all of the ports were started in 2015 (a couple were started later). There are no discretionary trades or revisions to these ports.

I converted every one of the ports to a sim as Nisser suggested in his post and compared the returns. There is no selection bias or removal of outliers. All of the ports were used in this study.

On average the sims performed better than the ports. The mean difference in annualized returns of the sims versus the ports was 8.6% in favor of the sims.

P123 says there is some look-ahead bias. My study supports this conclusion and puts a ballpark magnitude on this effect. People's anecdotal findings are not far from the 8.6% lending additional support to this study.

This number may serve as a ballpark estimate of the magnitude of the effect until someone does a larger study. People can--on their own--make an assessment as to how important this is to their own investing.

Going forward, comparing the sims and ports will likely have different results because P123 has been Snapshotting some of the data for the last 6 months.

Thank you Nisser, Brett, Chaim and RT for making me aware that the effect of look-ahead bias is not trivial at P123. Again, people can make their own assessment as to how important this is to their investing. Please email me if you want further details of my study or if you want me to send you a copy of the spreadsheet I used.

Best,

Jim

From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Oct 10, 2020 2:33:32 PM       
Edit 4 times, last edit by Jrinne at Oct 10, 2020 2:36:24 PM
yuvaltaylor
Re: I have some data (port vs sim)

Nisser, Brett, Chaim and RT,

I looked at 30 ports that are on automatic. Almost all of the ports were started in 2015 (a couple were started later). There are no discretionary trades or revisions to these ports.

I converted every one of the ports to a sim as Nisser suggested in his post and compared the returns. There is no selection bias or removal of outliers. All of the ports were used in this study.

On average the sims performed better than the ports. The mean difference in annualized returns of the sims versus the ports was 8.6% in favor of the sims.

P123 says there is some look-ahead bias. My study supports this conclusion and puts a ballpark magnitude on this effect. People's anecdotal findings are not far from the 8.6% lending additional support to this study.

This number may serve as a ballpark estimate of the magnitude of the effect until someone does a larger study. People can--on their own--make an assessment as to how important this is to their own investing.

Going forward, comparing the sims and ports will likely have different results because P123 has been Snapshotting some of the data for the last 6 months.

Thank you Nisser, Brett, Chaim and RT for making me aware that the effect of look-ahead bias is not trivial at P123. Again, people can make their own assessment as to how important this is to their investing. Please email me if you want further details of my study or if you want me to send you a copy of the spreadsheet I used.

Best,

Jim

What price were you using after you converted your live strategies to simulations? Previous close, next average of high and low, open, or next close? The default is previous close. If you didn't change that, that could easily account for a great deal of the 8.6% difference. In order to properly compare a live strategy to a simulation, you need to run it using the average of high and low and next close and take the average of those. Probably using just one of those is good enough. I'm also curious what the median difference is and how many live strategies out of the thirty you tested gave superior returns to the simulations.

Yuval Taylor
Product Manager, Portfolio123
invest(igations)
Any opinions or recommendations in this message are not opinions or recommendations of Portfolio123 Securities LLC.

Oct 10, 2020 11:00:43 PM       
Jrinne
Re: I have some data (port vs sim)

Yuval,

These are good questions. I get tired of the motivated reasoning on this site. Sometimes I turn into CNN and FOX news to get some relief.

But these are good questions.

I also hope that you appreciate that these are just my findings with little comment or discussion. Maybe not a lot of motivated reasoning from me (today at least). I do have some questions still as you may notice. I do not mean to imply that my little study is the final word.

I would have been happiest to find no difference. People can decide on their own how big of a difference this is. Maybe look at their own ports that have no revisions or discretionary trades.


Jim
What price were you using after you converted your live strategies to simulations? Previous close, next average of high and low, open, or next close? The default is previous close.

Correct. Previous close is the default. Is there a reason for that? If not, I wonder if that might not be worth changing. And obviously that would be a big problem for anyone not noticing. Should I make a feature request?

This is a concern indeed, and I was already aware of it.

That was the first thing I changed after converting the port to a sim. I switch the price to "Next Average of Hi and Low" and then I checked that the slippage was set to variable. As you know that is the default and I did not change the slippage for the sim.

In order to properly compare a live strategy to a simulation, you need to run it using the average of high and low and next close and take the average of those. Probably using just one of those is good enough.

I though so too.

I'm also curious what the median difference is


So again, excellent questions and much appreciated. Let me attach a copy of my results and you can enter the data into a spreadsheet and manipulate it if you want (getting MAD etc if you wish).

But in a word the median was: 8.5% (in favor of the sims).

Let me make a few simple numerical comments. Something like a t-test is not appropriate here. These are one person’s ports and they are not even close to being independent (the data is not IID and not even close). And 30 is still a pretty small sample which aggravates this problem. As an example of not being independent, some ports used the same ranking system.

Anyway I did not do that with statistics here. The mean or median, perhaps the range and maybe the inter-quartile range could be looked at and people can do that with (and whatever other manipulations they wish) with this data. Just do not accuse me of bad statistics: the median or mean are the best unbiased estimates we have.

...and how many live strategies out of the thirty you tested gave superior returns to the simulations.

So I sorted the data (see below) and it looks like the port performed better than the sim in 5 cases.

I highlighted one sim and port in peach where the sim and port were close (port beating sim). This was an SP 500 universe and did not use earnings estimates.

Anecdotally this seem to hold.

I am considering the possibility that one of my ports with no sentiment and a SP 500 universe might have some value to me.

But I would not mind understanding how good/bad the earnings estimate data is, specifically.

One last note: not too good on those ports that I developed 5 years ago. Although not necessarily worse than some of the desingers. Especially considering that they have almost universally removed their worst performing models. Olikea is an example of someone who is gone now and has stopped removing his Designer Models. Maybe there is less selection bias in his models. He is EXTREMELY savvy and intelligent: I hope he is doing well wherever he is. People should look there—before looking at my models—if the want an unbiased sample of a savvy investor’s out-of sample results.

Olikea has 24 Designer Models—all with at least 2 years of data with at least 2 years of excess returns. BTW, median of those 2-year excess returns: -34.17.

Am I judging now or am I just having a natural reaction when I say: Whoa! Knowing me, probably judging but to be fair I am having some palpitations, feeling some chest pain and wondering why I do not keep some nitroglycerin tablets around the house after seeing that number for the first time. That is something to think about.

Anyway, forgive me for my natural curiosity. I wonder how good those models (Olikea’s) would look now in a backtest.

There are more problems than look-ahead bias in my old ports (and the Designer Models).

One thing I find entertaining about CNN and Fox news is how the same person can change their opinion literally overnight and no one seems to notice.

The backtests were sooooo important to the members not too long ago. But I do not think that was ever your position, Yuval. I might be wrong (an please correct me if I am wrong) but you have use a lot of techniques that I would generally put into the category of cross-validation. You may disagree on the names but good techniques by any name, IMHO.

Now the cool new thing with members is…… Well, not backtests. Okay, got it.

Yuval, these are good questions. Today, after looking at CNN and Fox news I might come back to P123 for relief (reversing my usual weekend viewing habits).

Let me know if I can do some further manipulations on the data. And do not forget that some of the Designer Models have been around for 5 years if you want a larger sample with a more diverse group of people designing the models.

Thank you for your interest and excellent comments.

Edit: I am happy to share my data. But I wonder if you could looking at Olikea’s models using the following: "In order to properly compare a live strategy to a simulation, you need to run it using the average of high and low and next close and take the average of those."

No one will accuse Olikea of knowing ahead-of-time that we would be discussing this. He is a savvy investor which is good. There are no recent revisions in his models. And I think his models are pretty diverse with a decent sample size. I cannot do that and I am out of data.


Best,

Jim

Attachment Excel SpreadSheet.png (132869 bytes) (Download count: 71)


From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Oct 11, 2020 6:32:31 AM       
Edit 35 times, last edit by Jrinne at Oct 11, 2020 9:01:29 AM
yuvaltaylor
Re: I have some data (port vs sim)

Thank you, Jim, for answering my questions, and for providing the data. This was a valuable study indeed, and the results bear thinking about quite seriously.

I'd like to point out a few facts about this data.

First the correlation between the two is quite strong: 0.76. That's not so bad considering these systems were based on completely different data providers.

Second, the fact that the sims performed better than the live strategies could be due entirely to mean reversion since the live strategies performed so poorly. These are five-year live strategies, during which the Russell 2000 had a CAGR of 8.58%. Yet your live strategies had a median return of -9.92%. If they had a median return of +9.92%, do you think the simulations would have a median return of 18% or 22%? If you conclude that simulations tend to outperform live strategies by about 8.5% based on this data, that would be your conclusion. But it might make more sense, given this data, to conclude that simulations will vary significantly from live strategies due to different data providers, and if the live strategies perform very poorly, then simulations will perform better, while if the live strategies perform very well, simulations will perform worse.

To back this up, here is a scatterplot of your first and third columns, live strategy return on the X axis and the difference on the Y axis. If you ignore the outliers, where the live strategy return was EXTREMELY bad, the slope of the linear regression is -0.66. This is more or less what you'd expect: as the returns get higher, the difference gets lower. If you extrapolate from the trendline, you'd see that if your live strategies had returns of greater than 7%, they would likely outperform the sims. (I realize that by ignoring the outliers, I'm manipulating the data in favor of my point, but I always trim outliers when doing linear regression, and my eyes tell me that those lowest three or four X points don't really fit with the rest of the data.)

Jim, I think you might be entirely right that sims based on FactSet data will tend to outperform live strategies based primarily on Compustat data. It certainly merits further exploration, which I'll try to do sometime this week as time permits. But there is another possibility that explains this data equally well, and that is my hypothesis that sims will outperform live strategies when those live strategies perform very poorly and will underperform live strategies when those strategies perform very well, simply because of the principle of mean reversion.

Attachment live strategy and sim difference.png (10757 bytes) (Download count: 56)


Yuval Taylor
Product Manager, Portfolio123
invest(igations)
Any opinions or recommendations in this message are not opinions or recommendations of Portfolio123 Securities LLC.

Oct 11, 2020 8:56:30 PM       
yuvaltaylor
Re: I have some data (port vs sim)

Now I don't know if the following experiment will convince you that FactSet's not-very-PIT simulations aren't very different from Compustat's quite-PIT simulations, but here goes.

A while back I created 48 ranking systems based on the AAII screens.

I took the first thirty of those ranking systems in alphabetical order and created simulations based on them, buying the top twenty stocks every four weeks and selling when their rank position went down to 100, over the last five years, with a pretty wide universe in terms of liquidity. I then ran the sims using both FactSet and Compustat data.

I'm pasting the results below. You'll see the median advantage that FactSet has over Compustat is 0.19%. The correlation between the two sets of data is 0.66.

Now this is not the same as your experiment, I admit, which compared live strategies to simulations. But maybe you'll find this data useful. If not, I tried.

Attachment factset vs compustat sims.png (19208 bytes) (Download count: 53)


Yuval Taylor
Product Manager, Portfolio123
invest(igations)
Any opinions or recommendations in this message are not opinions or recommendations of Portfolio123 Securities LLC.

Oct 11, 2020 9:40:46 PM       
RTNL
Re: I have some data (port vs sim)

Now I don't know if the following experiment will convince you that FactSet's not-very-PIT simulations aren't very different from Compustat's quite-PIT simulations, but here goes.

A while back I created 48 ranking systems based on the AAII screens.

I took the first thirty of those ranking systems in alphabetical order and created simulations based on them, buying the top twenty stocks every four weeks and selling when their rank position went down to 100, over the last five years, with a pretty wide universe in terms of liquidity. I then ran the sims using both FactSet and Compustat data.



Thanks Yuval. That is helpful. Did you find anything insightful when looking at the outliers?

Regards!

Oct 11, 2020 10:00:51 PM       
yuvaltaylor
Re: I have some data (port vs sim)

I would like to add one more thought about the PIT vs non-PIT issue. The idea that non-PIT simulations would outperform live strategies in general must rest on some theoretical basis. The unstated hypothesis seems to be that data with look-ahead bias will give better results than data without look-ahead bias. That is certainly true of any process in which the data is taken into account prior to establishment of the strategy. However, in the examples we're looking at here, that is not the case. The strategies were established prior to looking at the data. So data that has look-ahead bias will offer no advantage over data that is not subject to look-ahead bias.

In terms of strategy development, it is optimal to limit look-ahead bias to a minimum. But in terms of testing old strategies, it should make no difference.

Yuval Taylor
Product Manager, Portfolio123
invest(igations)
Any opinions or recommendations in this message are not opinions or recommendations of Portfolio123 Securities LLC.

Oct 12, 2020 9:30:24 AM       
Edit 1 times, last edit by yuvaltaylor at Oct 12, 2020 9:32:03 AM
nisser
Re: I have some data (port vs sim)

I looked at all 6 live ports with the "average of hi and lo" and these are the results. All over the place but generally down.

Attachment Ports.jpg (160993 bytes) (Download count: 36)


Oct 12, 2020 1:53:37 PM