Index | Recent Threads | Who's Online | Search

Posts: 32    Pages: 4    1 2 3 4 Next
Last Post
New Thread
This topic has been viewed 3354 times and has 31 replies
RJJ
How many R2G's have INCORRECT PERFORMANCE Stats?

How do we know which R2G's have been affected by P123 data "fixes"? From my understanding performance stats are "locked" and may not be reflecting accurate information after a data fix has been made. Please refer to this thread.. http://www.portfolio123.com/mvnforum/viewthread_thread,8135_offset,0 . I don't understand how inaccurate stats can be useful. How do we know how many data fixes there have been over time and which models they affect and to what degree? Per this comment.. http://screencast.com/t/Sb4aUJUmd can we get some feedback from P123 as to how and when this can be addressed?

Dec 2, 2014 10:18:20 PM       
marco
Re: How many R2G's have INCORRECT PERFORMANCE Stats?

Rerunning a simulation of a launched r2g when there are subs will likely change the current holdings creating a lot of confusion/turnover for subs following the model.

Several designer take it upon themselves to rerun a simulation when fixes are announced, and report their findings in the model description.

As a customer you can/should ask the designer to run a current sim.

But in the end, out-of-sample is what matters and time will sort it out.

What we can say is that usually the fixes are not material to sims.

There is no easy solution and we're open to suggestions. We could, for example, periodically run sims for every r2g and report the stats in a separate tab called 'latest sim'

Thanks

Portfolio123 Staff.

Dec 3, 2014 9:32:22 AM       
davidbv
Re: How many R2G's have INCORRECT PERFORMANCE Stats?

Developers should be required to re-run simulations of their R2Gs on at least a monthly basis and report. I see a lot of R2Gs that have not been looked at for over 30 days. I have a feeling many developers just put them on set and forget. We all know things are changing over time and it is un-nerving to believe the model is not being tested regularly. This should include all the free ones.

David

Dec 3, 2014 10:49:55 AM       
aurelaurel
Re: How many R2G's have INCORRECT PERFORMANCE Stats?

@RJJ, what's the point? simulation's stats are useless for a customer. They are only useful in context, as a feedback process for the developer.

I'd like to be able to change the benchmark on existing R2G though. Many models including mine have the S&P 500 as a benchmark when we should have the Russell 2000. This is an oversight that would be good to eventually correct.

Especially so that out of sample alpha and beta can be standardized throughout R2G.

Dec 3, 2014 11:02:07 AM       
fips
Re: How many R2G's have INCORRECT PERFORMANCE Stats?

David,

I think there have been issues in the past with "last viewed"-stats. And even if designers have viewed it, it doesn't mean they have done so for re-evaluating the model. Rather, they might review their respective port (that one which models the R2G) much more frequently and try to improve it that way. But in general I agree that more interaction between the designer and the community/sub might help to keep models cutting edge. In the end it will be like with most mutual funds, it's a trade-off between transparency for building trust and secrecy to keep your "model ingrediants" private. OOS data will sort the wheat from the chaff.

Aurélien,
what rules do you propose to check for the benchmark? Thankfully we can roam free to build our own custom universes. I think it would hard to find the perfect match and it costs lots of time to examine the model. Maybe it would be easier to introduce a mandatory default benchmark, like the S&P 500. Fair enough, the choice will often not match the model, but at least it's as objective as it gets.

Best,
fips

Scared of drawdowns?
Annoyed by high turnover?

Dec 3, 2014 12:17:41 PM       
DennyHalwes
smile Re: How many R2G's have INCORRECT PERFORMANCE Stats?

A few comment about re-running R2G Ports.

There are other reasons to pick a benchmark besides the classical comparison of a mutual fund or ETF to it's similar index. Some of my Ports use the S&P 500 as the benchmark because it is used by the market timing rules. If I change the benchmark to reflect more closely the range of stocks the Port buys, it makes a BIG difference in the timing dates that I don’t want.

Also, some members want to compare Ports to the S&P 500 since it is viewed as the safer index for buy and hold comparisons.

Currently, a R2G designer can’t make any changes to a Port without “revising” it. That includes simply re-running it with no changes. If we do, it shows up as a revision. How do we allow re-running a Port and assure the designer didn’t “change” it?

Do we re-run the out-of-sample data also? Over time there will be data changes in that also. I see that opening a can of worms.

And how do we re-run the multiple revisions starting at the proper revision dates using the Sim that created the revision while incorporating them properly with the original pre-launch data and later revisions? Did the designer even keep a copy of all intermediate revision Sims? I see that as a bigger can of worms.

We must be careful what we ask for and/or require of the designers!

Denny cool

"It's amazing what ordinary people can do if they set out without preconceived notions."
- Charles F. Kettering

Dec 3, 2014 4:01:39 PM       
Edit 3 times, last edit by DennyHalwes at Dec 3, 2014 4:16:35 PM
aurelaurel
Re: How many R2G's have INCORRECT PERFORMANCE Stats?

@ Denny I used to use close(0,#bench) for volatility models but switched to close(0,getseries("$sp500")) for exactly that reason.

Dec 3, 2014 4:47:19 PM       
RJJ
Re: How many R2G's have INCORRECT PERFORMANCE Stats?

Marco, I think your idea of "periodically run sims for every r2g and report the stats in a separate tab called 'latest sim'" is a great idea. As a subscriber to P123 I would "expect" that to be done if changes are made in the background. I'm still perplexed as to why stats are locked and not updated with correct info to begin with. I don't know how you address a change in recommended holdings if sims are re-ran but the subs have a right to know the accurate data. If I'm holding XYZ corp and come to find out it didn't meet the rules based on "fixed" data it is not ethical to not disclose that info to me.

As for waiting for out of sample data to sort things out..????? Folks subscribe to R2G's with the trust the info they based their decision on to do so is accurate. If it turns out to not be accurate isn't that a huge liability to not disclose and update the new findings? Waiting until they lose a lot of money to only find out there was a "fix" in the data that affected the sim is not ethical. We appreciate your forth coming in fixing things as needed in the backdrops. We just need peace of mind that data that is put in front of us is the updated and accurate AFTER those fixes.

I do not think that putting the ball in the sub's court to "ask" the developer to re-run the sims is appropriate. Most subs do not even know there is reason to do so. If P123 makes changes that affect R2G's it should be P123 responsibility to disclose correct performance stats of all models at the same time. This needs to be done asap.

Marco, I have trust that you will get new stats disclosed soon. Thank you in advance!

Dec 3, 2014 5:04:35 PM       
primus
Re: How many R2G's have INCORRECT PERFORMANCE Stats?

The title of this post is leading and misdirecting. The ports' performances are NOT INCORRECT. Rather, they are out of sample. While it would be nice to compare the pro-forma simulation versus the out of sample, what was traded in the past needs to be written in stone.

"The world is. The world is. Love and life are deep maybe as his eyes are wide." - Rush, "Tom Sawyer"
http://the-world-is.com/blog

Dec 3, 2014 5:26:57 PM       
DennyHalwes
confused Re: How many R2G's have INCORRECT PERFORMANCE Stats?

I agree with Primus. From the beginning, R2G Ports were to be written in stone. I see an even bigger problem with rerunning the Sims than the ones I mentioned above. Many of them will inevitably end up holding different stocks. What will happen is a data item that was fixed 10 or more years ago will result in buying a different stock which in a few years will snowball into all different stocks bought at different times making different performance. That happens ALL the time I rerun Sims a year later.

We still don't have the actual open prices for stocks from 1999 to 08/27/2004. Hopefully, Marco will up date that some time in the future as he promised. What will that do to R2G Sim reruns?

Are those of you wanting to rerun the Sims willing to sell all current holdings, buy new stocks, and explain to your subs why you screwed up their portfolio? I don't think there is enough reflection in this thread on all the problem that rerunning the Sims will cause. This WILL be a can of worms.

As I said, be careful of what you ask for!

Denny cool

"It's amazing what ordinary people can do if they set out without preconceived notions."
- Charles F. Kettering

Dec 4, 2014 1:13:20 AM       
Posts: 32    Pages: 4    1 2 3 4 Next
 Last Post