A couple of days ago, I started to another thread (“DEVASTATING DISCREPANCIES?”) on the same subject. However, I think this issue really needs renewed attention from Marco and the staff, so I am starting a new thread so that this post doesn’t get lost in the jumble.
After several posts in response to my Devastating Discrepancies thread that supported the idea of ‘data perturbances’ as a good thing, I decided that I needed to push back a little on the implication that I was expecting too much from the data - that I should embrace BAD DATA as something helpful in strategy design.
The two people involved with the portfolios discussed here (including me) have a combined quarter-of-a-century experience with Portfolio123 system design. That means we joined this website not long after Marco launched it. We are both well-aware of the challenges faced by strategy developers and mindful that designers must always try to avoid trip-wires along the way that can make in-sample results far different from out-of-sample results.
As I discussed in detail in my earlier posts in the other thread, many of the figures generated by the P123 system are flat INCORRECT. I stand by that statement. Do you really want to abandon an investment approach (as Yuval suggests) when P123’s system is giving you bad data? Wouldn’t it be better to ask P123 to fix the problem so we can build strategies with accurate data?
I was trying to prompt the P123 staff to sort out these errors. Apparently, not many people paid close enough attention to the details of all the investigative work I did. If they had, they would see that the data is ERRONEOUS. I don’t believe we can successfully develop useful strategies with BAD DATA. You know the old acronym: GIGO (garbage-in, garbage-out).
Rather than make claims, I will (again) show a handful of incidents and allow the data to speak for itself. Hopefully, my point will get across this time. For simplification, I re-ran the same portfolio again today (portid=1509975) with $0 transaction costs and 0% slippage. The strategy is very conservative and uses just two, long ETF assets, the S&P 500 (SPY) and the 1-3 yr Treasury (SHY) as a proxy for cash. The sim ran from 01/02/2000 - present. The “Price for Transactions” tab is set to “Next Open,” as shown in this graphic:
The Transaction results of the Sim are still INCONSISTENT WITH FACTS regarding what transpired in REAL TIME. Dividend amounts are still incorrect, but I will focus this analysis on the INCORRECT PRICES quoted to purchase the equities because the many thousands of dollars in difference-per-transaction cause dramatically different results than the expected results had the simulation used correct data.
Specifically, I will focus ONLY on the listed purchase price for the S&P 500 SPDR (SPY), one of the most-traded securities in the world with more than 60 million shares traded daily. SPY is not a thinly traded penny-stock with questionable pricing. The table below shows a screenshot of the Transactions in the first two years after the portfolio was launched, from 12/31/99 - 08/05/02. I have identified five instances of purchases of the full amount of shares of SPY (not the small-but-pernicious “Buy-Sell Difference” transactions that persistently haunt our portfolios). Outlined in red are the Transactions in question:
Recall that I set this simulation to use Opening Prices (without commissions or slippage) for this exercise. The table below shows the transactions as listed by P123 in the second column, accompanied in the third column by the CORRECT OPENING PRICE for SPY on these dates. Look them up for yourself if you question the veracity of this information.
The fourth column shows the percentage discrepancy between the actual price and the prices listed by P123. The shares then multiply these figures, and the calculations for P123’s INCORRECT amounts (sixth column), then the CORRECT AMOUNTS (seventh column), and the far-right column shows the difference in dollars between P123’s incorrect quantities and the CORRECT amounts. As you can see, it would add up fast! And these numbers come from early in the portfolio’s history - before the number of shares increase 10 or 20-fold!
>>>>Interesting sub-fact: For the first two purchases of SPY, P123’s calculation is erroneous (even using the incorrect price). For example, the opening purchase of 687 shares x $145.44 = $99,917.28 while P123 lists $99,915.56. I didn’t look further, but I wonder how often miscalculations like this occur? It seems as if computers should be competent at the basics of multiplying two numbers if the code is carefully crafted.
CHANGE IN THE DATA: I did not go through all 250 transactions to see when they finally became correct, but at the top of the Transaction page, the final three Opening Prices for SPY and Total Amounts are CORRECT. This table below shows the comparison data:
This screencap of the Transactions in 2016 confirms these numbers:
It is evident that P123’s data engine is capturing INCORRECT PRICES for transactions early in the run and then somewhere along the line, in the course of the last 17 years, it gets back on track. I hope that this analysis is taken seriously and P123 management will take it upon themselves to identify the bug and correct this error. I do not see why members, some of whom have paid many thousands of dollars over the last 13 years, should have to consider and compensate for these fundamental errors when designing and using portfolios. I would prefer to introduce random perturbances at my discretion to test the robustness of my investment systems after they are developed, not always having in the back of my mind that my results aren’t correct from the start!
Don’t get me wrong; P123 has - by far - the best product available in this niche of the investment market. The product we get per dollar is an incredible bargain. I am just advocating for accurate prices if it is available. I suspect that what I have identified here is probably a bug in the system. The Opening Prices are incorrect at the beginning of the portfolio, but then are correct by the end. To me, that says there is a line of code that is amiss somewhere along the way. I am trying to assist P123 management as well as my fellow users by pointing out this issue so that something can be done to find and correct the source of this error so we can obtain greater accuracy and repeatability.
The investment business runs on numbers. The only other areas of life that use figures more actively are perhaps physics and, well… baseball. Because of the number-intensive nature of the investment industry, I feel that we SHOULD be able to work with CORRECT DATA - at least the correct prices per share of stock - when we pay a sizable amount of money every month for this service. It doesn’t seem like too much to ask - or maybe I’m wrong. What do the rest of you think?
–Chris
P.S. - I just discovered with a little more investigation of the prices of SPY, that the $ per share used by the P123 system in the early years are from the CLOSING Prices for the security - not the OPENING Prices that I set. Then later in the run, it is accurately using OPENING Prices. This makes me almost SURE this is a coding error that can be corrected - it seems less likely that it’s a quality-of-data issue from the vendor.
I checked the prices of the holdings in this portfolio when it was built several months ago to confirm to my satisfaction that the Sim was correctly using Opening Prices for purchases and sales and that those Opening Prices were correct (they were).However, checking these prices reveals that many (if not most) prices are incorrect. My guess is that something went awry in the coding when adding the Rebalance Module recently. I believe this problem is fixable!
