IMPORTANT: SP estimate series were re-built due to recent fix

Dear All,

We’ve rebult the “Fed Model” series: #SPEPSCY, #SPEPSNY , etc due to recent fix discussed here

You can see the image below for an idea on the magnitude of the changes. It should not affect your systems much since the overall trends are the same. For example in one of my systems I entered a hedge one week earlier with these new values

Very sorry for the inconvenience

NOTE: please notice that in the most recent data there are no differences since the bug fix had a compounding effect the older the point in time date.


The link https://ww.portfolio123.com/mvnforum/viewthread_thread,8134 is not working for me. It should be http://www.portfolio123.com/mvnforum/viewthread_thread,8134

Thanks!

How can we be informed of the “exact” before and after changes in performance when a change / “fix” is made? When we subscribe to models we are under the assumption the data is correct. Shouldn’t we be informed as to exactly how the models were impacted? Is the main page of the R2G models automatically updated to the new stats or is it still reflecting the info from prior to this fix?

I think the impact on R2G models from changes to P123 data and fixes should be investigated. In particular older models could be affected a lot. Perhaps all R2G models should be converted into sims and re-run at periodic intervals to provide updated performance figures in line with most recent P123 data. Currently historic R2G performance is frozen and not updated due to “fixes”.

Thank you for posting that George. I’m surprised more folks don’t have a raised eye brow to this. If the historic data is indeed locked down on the main R2G model screens shouldn’t there be a big red disclaimer that states “THIS DATA MAY NO LONGER BE ACCURATE DUE TO FIXES MADE ON X-X-XX DATE”? How would subscribers feel about seeing that? It would make me think “what is the relevance of inaccurate data?” Perhaps the data is still very close to being accurate but how do we know? What about all the folks that don’t see notices of “fixes” in the community? I would assume that once errors are fixed the data of the models would automatically be reflecting the new stats. I have no idea how many fixes have been made to P123 data that may have affected the stats of the models.

Very concerned with reliability of accurate data and model performance stats.

I disagree with geov. P123 should embrace bug fixes and improvements to data quality. Making a big to do over a minor fix disincentives it from doing so. Besides, if a minor change in a single parameter causes a disproportionate change in performance, then the whole model was fragile in the first place.

Primus, there have been a long list of “fixes” over time, not all of them minor. I remember when the short interest ratio was “fixed” it caused havoc to sims. All those that had this in the ranking system were affected. Of course, bugs must be fixed, but it is not fair to R2G subs to provide outdated performance figures.

I have first hand experience of this. Middle of August 2014 I inadvertently converted the R2G portfolio of Best10 into a sim and ran the simulation. (This is now not possible anymore, because P123 fixed this.) After this the model included all the “fixes” to that date. Performance figures were adversely affected. So at least for Best10 the performance figures are up-to-date now.

I understand that users should be able to tell the difference between hypothetical performance. But what about real performance? You can’t just go changing a port’s forward looking historical performance. Out of sample performance is perhaps the only unbiased indicator of skill we, as quant investors, can latch on to.

Why can’t p123 implement a function which copies a port’s rules over to a sim for direct comparison? The trick would be to keep the rules and ranking systems private (proprietary).

Perhaps it should be a mix,

  • After every change or even simply periodically if possible just in case, Historical data (pre-launch of R2G) should be re-run (if it is not already)
  • OOS data / data after the launch of the R2G should remain as is, because those were the signals sent to subscribers, no matter what the bug/changes are.

I agree that altering past results diminishes transparency and it’s confusing to see graphs that would have shown different holdings that alter from your actual past trading signals.
The most transparent solution would be to have both point-in-time statistics to not deviate from previous trading signals and up-to-date stats. Depending on the data updates, they can indeed cause significant change in performance that I am sure most designers and subs would be eager to know about. Data presentation would inevitably become more extensive, but it would provide the greatest transparency (and is similar to what standard setters deem appropriate for financial statements in similar cases).

Georg, you said
Middle of August 2014 I inadvertently converted the R2G portfolio of Best10 into a sim and ran the simulation. (This is now not possible anymore, because P123 fixed this.) After this the model included all the “fixes” to that date. Performance figures were adversely affected. So at least for Best10 the performance figures are up-to-date now.
Do you mean you converted an R2G to a sim, re-ran it (now including more up-to-date and accurate data) and then the stats “transferred” to your R2G stats? Or were you referring to the stats in your personal sim/port?

Best,
fips

fips, I was not referring to my personal sim/port, if was referring to the R2G Best10. See the August 20 comment of Best10. The holdings also changed a bit with a stock appearing that was not bought before. I think the annualized return came down by about 4%.

Alright. Sorry, I was wondering about the exact procedure on how to manually update an R2G for new data.
I missed your comment in the parantheses that this is no longer possible.
Too bad, I would like to be able to do so, if changes would be displayed transparently.

Best,
fips