DEVASTATING DISCREPANCIES?

Research
1

I recently discovered some incredibly substantial differences between some of my Live Portfolios that I created in July-August 2017 and duplicate Simulation copies that I made from the live ports. I am trying to determine the source of the problem, and I am wondering if I am the only one experiencing this issue (hopefully). If no one else is seeing it, then it is something internal, and it can be resolved – or if everyone is experiencing it, then the p123 platform must the source of the disparity.

As an explanation of this issue, a member of the P123 staff told me, “We made a change to the timing of the award of cash dividends in simulation, and it looks like that is what caused the difference.”

While I give 100% support to Marco and P123’s efforts to make this site an accurate representation of past prices and conditions, it is frustrating when “adjustments” to the data consistently result in dramatically different (almost always worse) sim/portfolio performance.

Having been a subscriber to P123 since 2004, I know that it is unreasonable to assume that a re-run of an older portfolio will produce the same results as it did when created. Compustat, in particular, seems to always be doing corrections and refinements to their historical data. These changes are essential to establishing that the work we’re doing is as accurate a representation of point-in-time results as possible.

However, these aren’t “old” portfolios; they were created just 2-3 months ago! I think this problem is something very different because of the enormous difference in returns.

I discovered the issue when making simulation copies of my portfolios, with some small and some very large. It’s the substantial differences with a reduction in return of -50% or -75% that is causing me to lose sleep at night.

Here is a screen capture of a portfolio that I saved as a live version in August 2017 - just three months ago. It has an annual return of 36.20%:

This image shows an exact copy of the same portfolio, run on the same dates, made yesterday. It has an annual return of just 21.26%. The total difference at the end of 10 years is what is most troubling - $2.05 million - a loss of 72% in total return!

Also, these portfolios do nothing fancy related to the revised “Rebalance” module - they just use standard “% Portfolio Weight.”

Please re-run some of your recently created live portfolios as duplicate copies and see if the results you get are a dramatically bad as this example (I have others where the loss of Total return is about 50%). Report your results here if there is a substantial, unexplained difference.

Thank you,

Chris

2

I see that Aaron recently posted a thread titled, “Portfolio performance now includes accrued dividends.” Another p123 staff member told me that this - the dividend change - was the source of my discrepancies. It seems to me that adding in accrued dividends would perhaps improve the performance of portfolios rather than the dramatic decreases I am getting.

Nevertheless, I find it hard to believe that I could lose -72% of a portfolio’s return - or more than -$2 million over the 10-year life of a portfolio just because of adding in some accrued dividends.

I continue to hope that the cause of the problem is something inherent to my systems rather than a platform-wide change. However, check out re-runs of your recently created portfolios, folks. You may discover you have lost more than half the performance you thought you would get.

Chris

3

I don’t find large differences like this.
Have you set the simulation price to “Next Close” ?

4

Thanks for thinking about this, Georg. However, I use “Next Open” on all ports and sims.

I find that “next open” is closest to a price I can get in real-life, especially if I put in electronic market orders on ETFs anytime before the open (I enter all my orders on Sunday). It is virtually impossible to buy stocks or ETFs at “next close” and match the actual close price in real life. Miss it by a couple of seconds and you don’t get the closing price at all (you may get the next day’s price), not to mention that it is impossible to buy or sell multiple positions all at the same moment near the close.

My experience with allowing brokers to execute ‘At-the-Close’ orders on my behalf results in prices all over the board.

Is there some reason that you advocate “Next Close?” If so, don’t your actual results differ from your model?

Chris

5

Chris, did you just delete the live portfolio you reported this issue on ?

6

Hi Marco,

No, I haven’t deleted any live portfolios. I’m glad to know you are looking at this, though.

The portfolios I would most like you to review are this live Portfolio:

https://www.portfolio123.com/port_summary.jsp?portid=1504458

and this Simulation copy of it:

https://www.portfolio123.com/port_summary.jsp?portid=1502612

Note that these are a different set than the ones discussed to start this thread. However, this set is perhaps the most important one to me.

The difference between the two is about 4.5% per year, but over the life of the portfolio this adds up to a Total Return difference of $4,579% (Live Portfolio) compared to 2,387% (Simulation copy made yesterday) or about $2.2 million, which is a loss of about HALF the return.

Thank you,

Chris

7

The change in the dividend payment caused a change in the first few months of the simulation. Once you get that single difference, the effect is going to snowball as the simulation continues over time. That’s what happened here.

I think it’s important to point out: You have a single holding. Any change at all has the potential to cause huge swings, and an early difference will only magnify that effect. Couple that with a strategy that, effectively, changes from long-to-short and you get another increase in potential effect.

8

Paul, Aaron, and Marco,

So your last paragraph above would imply that anyone who is using a single-ETF portfolio should be aware of the possibility that their return could be devastated by 50%, 70% or more (as mine are)? If my portfolios hold 1, 2, 3 or more positions, the performance of all of them will be changed to some degree since they will all be getting these ‘accrued dividends’ I presume.

I guess I do not understand the implications of P123’s recent change to dividends. In another thread posted today, Aaron said:

“Moving forward, the computation behind performance for live portfolios, live books, and simulated portfolios will account for accrued (unpaid) dividends, decreasing apparent volatility and improving the accuracy of both the performance curve and various risk measurements.”

With due respect, since 2004 I have been monitoring the real-time performance of my investment portfolios that are based on p123’s ports and sims. For the most part, they are very accurate and coincide with what is obtained on paper (here, on p123). Entry and exit prices match the opening price on the day of the transaction (my setting) and the dividends are paid on the day and in the amount represented in the p123 ports. However, now that the strategies are going to be modified by these phantom “accrued dividends,” I don’t see how the p123 ports can match “real-life” performance any longer.

“Accrued dividends” are a liability that goes on the company’s books. They are NOT something that directly affects the prices and performance of an investor’s portfolio. Please explain this to me and all p123 members and how specifically it changes our portfolio’s returns. I have been involved in the investment industry for more than 30 years and I have never heard of a portfolio being “adjusted” by an accrued dividend.

If I have this all wrong, I apologize. But I am wrong because this change needs to be explained to me and all p123 members in detail because of the performance and accuracy implications going forward. Thank you.

Chris

9

Another thought:

All of my portfolios that have been affected are ETF-based. Please tell me, is my position in SPY is going to be affected by the accrued dividends of all S&P 500 companies that declare a dividend but have not yet paid it?· Is that what happened to the portfolios above? Where can I find documentation of the portion of each S&P 500 company’s accrued dividends and how it is affecting the price of my holding in SPY?

Put bluntly: I don’t get it!

Chris

10

Let me lead with my bad. I went in and looked for the dividend difference that I spotted yesterday, and I couldn’t find it. Sorry about that. I’m not sure what past Paul was looking at.

What I did find today is an errant Buy/Sell Difference in the market timing portfolio on 4/30/2001. Aaron said that he reviewed the transactions and found other, similar discrepancies in both portfolios. In my opinion, that probably means that the differences are related to changes in the rebalance/reconstitution tab, but that’s just a hunch. Given that we’re talking about runs made months ago, it could be related to any of a plethora of minor tweaks and bug fixes as well.

For the ETF Equity Rotation portfolios, the simulation simply chooses a different holding in the second week for some reason based on your custom series as they existed a few months ago. That’s enough to make the runs completely different.

The 50% return differences, though, are not caused by differences in the simulation, but rather because of a lack of diversification. If you only pick one stock or ETF and then you pick a different stock or ETF for whatever reason, then the returns are (probably) going to be massively different. At a minimum, unpredictably different. Remember, your market timing portfolio is effectively going long or short depending on what it’s picking. If you zig when you should have zagged, then 50% might be a small difference.

That’s really what it boils down to. In a portfolio with more holdings, the effect of a different choice here and there is likely going to be relatively minor. In the case of a one-holding portfolio, it’s huge, because you aren’t getting any risk mitigation through diversification.

11

In comparing the two charts I would look at the transactions between 1/08 and 1/10. The charts mirror each other except for this time frame. The hit taken in the sim copy in 2009 is what I imagine created the return difference. Your compounding would be much lower. You can see at 1/12 that the sim copy is already half the return of the live port.

The observation that really caught my attention is that the benchmark charts are not identical through this same time frame. I would think the benchmark charts would be identical regardless of whats going on in the ports… Though this discrepancy is most likely due to scaling…

12

Paul and P123,

Thank you for responding to me with the updated explanation. I’m glad to learn that P123 isn’t trying to break new ground by making us pay for accrued dividends out of the portfolio’s capital. haha

However, I have done some further examination of my own and am seeing consistently troubling transaction details that have the potential to be highly pernicious. Like your first explanation, I don’t think the second explanation you have provided covers these issues, but I will be glad to listen and learn if I am wrong.

As I mentioned before, these issues are seen in all my portfolios that were created a few months ago and then re-run today as exact simulation duplicates of the originals. There are small discrepancies between the two that are very devastating to total returns when added up over a 10-year or 17-year history. I would just like to get to the bottom of this, find out if there’s anything I can do to correct this situation or alternatively, let the Portfolio123 staff know that there is something very wrong so it can be corrected.

Specifically, I can see a significant difference between what P123 is showing for documentation of real-time transactions and what other sources (such as Yahoo Finance) are showing on the same transactions.

NEW, DETAILED REVIEW OF TRANSACTIONS

This is a new review of another portfolio (with its Simulation-Copy made yesterday)accompanied by more detail. To simplify the review from a 5-ETF universe, I am now looking at a different portfolio (all of my ETF portfolios are askew) that utilizes only two ETF positions: The S&P 500 (SPY) and the 1-3 Year Treasury Bond (SHY). This portfolio is a very straightforward market timing strategy. When conditions are satisfactorily bullish, it holds SPY. When conditions turn sufficiently bearish, it holds a cash proxy, i.e. SHY.

The original return of this portfolio (portid=1451777) shows a 12.79% CAGR from 01/02/00 - 11/08/17 with a total return of 756.76%.

Chart 1: ORIGINAL PORTFOLIO

Chart 2: DUPLICATE COPY MADE TODAY

A duplicate copy of the portfolio made today (portid=1510084) shows a 10.83% CAGR and a total return of 527.23% (-229.53% less).

I will narrow this down by only reviewing the portfolio’s transactions from its launch in January 2000 through to the end of April 2001 (only 16 months). In the original Live Portfolio (portid=1451777), there are fewer transactions than the Simulation Copy (portid=1510084) for this period and transactions are occurring on different dates.

Chart 3: ORIGINAL PORTFOLIO TRANSACTIONS

Chart 4: TRANSACTIONS FROM SIMULATION-COPY MADE TODAY
A duplicate Simulation-Copy of the portfolio made today shows incorrect amounts for the dividend payments and does not include several dividend payments within the first 25 transactions!

ANALYSIS

During the period examined, from January 2000 to April 2001, the ORIGINAL PORTFOLIO shows dividends paid three times: On April 28, 2000 (0.371/share), July 31, 2000 (0.348/share) and October 31, 2000 (0.375/share). I can see that these records show the correct dates paid on the far-right side column, but it does not show all of the dividend payments during this span of time.

According to other data sources, dividends on SPY were paid FIVE TIMES (not three) during this period: On March 17, 2000 (0.371/share), June 16, 2000 (0.348/share), Sept. 15, 2000 (0.375/share), December 15, 2000 (0.411/share), and March 16, 2001 (0.316/share).

Furthermore, SIMULATION COPY shows dividends paid on the correct date, but NOT IN THE CORRECT AMOUNT! For example, the Simulation Copy shows the first dividend paid (March 17, 2000) was 0.37 per share times 684 shares = 253.63. However, doing this multiplication would result in a dividend payment equal to 253.08 - 55 cents less than what is shown in the records. Also, the correct amount according to multiple data sources is 0.371 per share (not 0.37). If we multiply 0.371×684 shares = 253.764 or $0.134 more than what the Simulation shows.

The second dividend paid in the Simulation Copy is off by even more! P123 shows a dividend for SPY on June 16, which is the correct date. However, P123 shows the amount to be 0.35 x 685 shares = $238.65. If we do this math, the number should be $239.75, a difference of $1.10. Even more confusing, the actual amount of the dividend paid to shareholders of SHY was 0.348/share (not (0.35/share). The actual dividend paid on 685 shares of SHY on June 16, 2000 should be $238.38, which is -27 cents less that what P123 shows.

Neither my P123 Original Portfolio nor the P123 Simulation Copy has the correct dividend amount for June 16.

If it were just a couple of dividends, I could overlook this issue. But unfortunately, its not.

EXAMINING MORE DIVIDENDS

On December 15, 2000 and March 16, 2001, my P123 Original Portfolio shows NO DIVIDEND AT ALL! The Simulation Copy shows a dividend on both these dates, BUT THEY ARE THE WRONG PRICE. For example, on March 16, 2001, P123 shows a dividend for SPY of $0.32 per 706 shares = 222.74. But if we do the multiplication, that number should be $225.92. P123 shorted me $3.18. (This was at the start of the portfolio, 17 years ago.)

You can see that Paul was right, these small transactions get progressively larger as the portfolio progresses in time and we are only 1/10th of the way into the portfolio’s time span.

However, documentation from other sources shows the ACTUAL dividend paid was $0.316 per share (or $223.096 in my case). Even though the calculation is closer than before, I would prefer it to be correct and not just close.

I went through many calculations of dividends and through the life of the portfolio and so far, I have yet to find one that is correct! The mis-payment of dividends affects the total shares the portfolio could own each time it makes a purchase and the effect of compounding magnifies all of those small errors into VERY BIG ERRORS.

The difference caused by all these dividend miscalculations had my ORIGINAL PORTFOLIO purchasing 3,290 shares of SPY at the end, while the SIMULATION COPY only owned 2,409 shares (27% fewer) and have a total return that is 27% less.

Chris

13

MORE ERRORS FOUND - THIS TIME IN PRICES

Unfortunately, the errors I am finding do not stop with discrepancies in dividends. I also discovered that there are problems with the purchase and sale price of ETFs within the portfolios compared to the actual prices quoted by multiple stock-data providers. I won’t even go into to the prices issues shown in my Original Portfolio (they are even more severe than these).

The following pricing discrepancies are shown in the Simulation Copy that I ran today that is an exact copy of a Live Portfolio created a few months ago. They are from the last 18 months when my portfolio made either significant purchases or sales of the S&P 500 ETF (SPY):

Note that on none of the days above did the purchase or sale price-per-share for SPY in the Sim match any ACTUAL price recorded that day (open, close, high, low) by a number of other stock-price websites (such as Yahoo Finance and StockCharts). For simplicity, I excluded calculations of commissions and slippage. These are just the prices for SPY on P123 vs. prices for SPY quoted elsewhere.

Here is a screen capture of the prices for the last 18 months of my simulation:

Remember, these are from just the last 18 months of a 17+ year portfolio performance. There are a lot more where these came from. In fact, I did not see a single transaction that was not erroneous. Obviously, as Paul noted, the erroneous calculations for both dividends and purchase/sale prices are ballooned by the magic (or in this case, deleterious consequences) of compounding.

Yes, I agree 100%. However, it was not a “single difference.” The errors are in every transaction I looked at and you are right - every one of them snowballs and that is what causes $2.2 million differences!

This is very simple math. We are not splitting the atom here or sending a man to Venus. Moreover, this is P123’s bread and butter. I would think that getting the basics such as accurately quoting opening stock prices or accurately calculating dividend amounts would be high on the list of priorities. I have to assume that this is an error that occurred deep in the bowels of the P123 calculations when changing some other issue and it can be easily resolved. I certianly hope so, at least.

If I had not spent the hours I did to investigate these discrepancies, I’m sure that things may have proceeded along for who knows how long without these serious issues ever being addressed. However, I knew the differences I identified couldn’t be caused by a simple, minor, single instance. This is consistent errors in BOTH PRICING and DIVIDENDS that SPAN THE LIFE OF OUR PORTFOLIOS. I am sure that virtually every other member can find these same problems if they look as I did.

This is a SYSTEMIC PROBLEM that I hope will be addressed by P123 management ASAP so members can get back to designing blockbuster portfolios that are [b]consistently accurate[/b]. Please let us know when the problem has been located and fixed so we can get back to work!

Thank you,

Chris

14

P123 is rounding the display of dividend payment/share. The actual amount was $0.370823. So 684*0.370823=$253.64. P123 is off by a penny.

I don’t think there’s a problem with the payment amounts. The missing payments seem to be the problem.

Walter

15

Hi Walter,

I took into consideration that P123 is rounding the dividend amounts to 2 decimals. The dividend quote you mentioned still shows an inaccuracy (even if it is only a penny on that one) and other dividends are incorrect by much larger amounts. Plus, as you noted, there are transactions appearing and transactions missing in the portfolios, depending on which one you look at.

This was my conclusion after staring at dividend details and comparing P123’s to other sources for several hours:

Every dividend I looked at is incorrect!

Furthermore, what I said my second post on this page is more deleterious to the accuracy of our portfolios:

All the ENTRY and EXIT prices that I looked at for SPY are INCORRECT!

You are right Paul; the first dividend payment that you saw that was wrong. However, it is also the errant “Buy/Sell Difference” that is wrong.

The problem is that you didn’t go any further after seeing those first errors, both initially on the first dividend you observed and Thursday on the first “Buy/Sell Difference” you saw. If you had looked further, I think you would have seen what I did: that MOST TRANSACTIONS ARE INCORRECT. I may even go as far as to speculate that EVERY TRANSACTION IS INCORRECT because every one that I looked at is wrong.

You said that Aaron “reviewed the transactions and found other, similar discrepancies in both portfolios.” But he went no further than that? “Hmmm… lots of problems - oh well.” Please tell me that is not what happened!

Paul, you went on to say; [quote]
“In my opinion, that probably means that the differences are related to changes in the rebalance/reconstitution tab, but that’s just a hunch. Given that we’re talking about runs made months ago, it could be related to any of a plethora of minor tweaks and bug fixes as well.”
[/quote]

The run was actually made on Thursday, November 8, not months ago. And regardless of when they were made, isn’t it appropriate to find out why an investor who has been a member of P123 for more than 13 years and has brought to your attention a portfolio that is now showing a $2.2 MILLION discrepancy deserves some deep investigation? (Instead of a response of “oh well, probably problems with the new rebalance section.”)

I also wondered in the last few days if this issue is related to the revised “REBALANCE” module since I don’t ever recall there previously being these widespread problems with basic pricing. I hope that Marco, the P123 staff, and P123 members will conclude along with me that this is an unacceptable situation. I hope you guys don’t just move along from here without addressing this serious problem.

Please understand, I am not trying to be a jerk here. I am pushing hard because I want you guys to take this as seriously as I am (and I’m sure other members, also) and I don’t think that is happening so far. I want Portfolio123 to succeed!

Problems like this, if not resolved, can become devastating to a small business if the word spreads that there are widespread inaccurate-data problems - and when notified nothing was done about it. Nobody wants that to happen. I hope P123 is here for decades to come to come and continues to thrive!

I look forward to learning on Friday how this will be addressed and hopefully it is something easily corrected for you guys. Thank you again for your efforts to provide investors with the best (and hopefully most accurate) investment tools possible!

Chris

16

Hi Chris,

I don’t normally run ETF sims, but with stock sims if both transaction costs and slippage are set to zero, then the ‘Tot Fees’ column goes to zero. Could you check that those items are set to zero.

Walter

PS I checked the 06/16/2000 div and it’s right, too (minus a penny). If there’s one that is definitely, positively wrong, let me know and I’ll check it too.

17

Walter,

No transaction costs are included in the portfolio calculations I provided in this thread. I checked all the transactions discussed above in this thread and they are all wrong. Essentially, pick the price for any purchase or sale of the S&P 500 ETF (SPY) and the price is incorrect, as I documented. Pick a dividend payment, and it will be off slightly.

In other words, they are all " definitely, positively wrong." haha

Thanks much. I’m hoping Marco and the staff will sort this out tomorrow by fixing their errant code.

Chris

18

i:

I don’t know for sure if what you’re reporting is truly an error or not. Hopefully, P123 will address it. I want to thank you for your post.

Bill

19

Hi Chris,

The 11/21/2016 SPY sim price, $219.59, is correct if you have ‘Price for Transactions’ set to ‘Next Average of Hi and Low’. I just ran a sim this morning and P123 posts the correct numbers for ‘Next Average of Hi and Low’ and ‘Next Open’.

And why are there data in the ‘Tot Fees’ column? In my sims, setting slippage and commission to zero, makes that column zero, too.

Walter

20

RE: prices & dividends. I don’t think there’s an issue.

For prices take SPY 11/21/16 bought at 219.59. On that day the “as-reported” hi & lo were 220.18 & 219.00 = 219.59. When you look at historical prices you need to make sure you are looking at “as-reported” (the price on the exchanges) with future splits AND dividends reversed out (by default P123 & stockcharts show adjusted prices series in charts so it looks smooth).

For dividends … you need access to the full precision. Seems ok for the spot checks I did. And I don’t think there are missing SPY dividends either. SPY is being bought & sold and I don’t think it was held prior to ex-date of the the two “missing” dividends.

Lets not mix everything together or the discussion will get out of control.If you think there’s still a problem with sim prices and/or dividends lets do that on a different thread. The issue that started this thread is a different one. The initial investigation was inconclusive as to why around the beginning of the sim a different ETF was bought. To confuse the matter there seem to have been some changes in the InList. We’re still investigating this.

Thanks