ETFReplay vs Portfolio 123, different backtest results

I used ETFReplay and recently started using Portfolio123.

I am trying to replicate the ETFReplay model to Portfolio 123. However, I found the results in ETFReplay backtesting quite different from Portfolio123.

In ETFReplay, I have a portfolio with four ETF (IMW, MDY,QQQ, SPY). I picked the top performer based on past 3 months return on a month rebalance, with a cash filter. This means, that whenever the top performer’s 3 month returns are worse than that of the cash (based on “SHY”), SHY will be bought instead of the top performer.

Based on ETFReplay, the cumulative return from 31 Oct 2002 to 16 Feb 2018 is 654.2% with volatility of 15.5%.

I tried to replicate this in Portfolio 123 with the following portfolio:

Universe: IMW, MDY, QQQ, SPY
Ranking: close(0)/close(90), the higher the better
Buy rules: rankpos=1 and close(0)/close(90)-1 > close(0,getseries(“SHY”))/close(90,getseries(“SHY”))-1.
Sell Rules: rankpos>1

Portfolio123 shows a return of 106.49% with a maximum drawdown of -54.7%.

The result of Portfolio123 is so much different from that in ETFReplay while the two models are almost the same (with one difference that the Portfolio123 model’s cash is in cash, not in SHY).

  1. Did i do anything wrong in my portfolio123 model?
  2. How can I replicate the ETFReplay model in the Portfolio123?

many thanks

B

Hi brwg,

For ranking, Try ranking based on close(0)/close(63) - the input for close is # of bars (not days) and 63 would be approximately 3 months.

I’m not sure about the buy rule. It’s saying only buy when close(0)/close(63) > same for SHY, but if that condition is not met nothing would be bought (if I understand correctly). Not sure if that’s intended, or if SHY should be available in the universe itself to be eligible to be bought. I don’t think the rankpos=1 is needed though in that case (if it’s in the eligible universe) if the intent is to buy SHY when it outperforms everything else over trailing 63 bars…

edit: (I guess I think if the universe is expanded to include SHY, if it’s the top ranking etf based on the ranking system close(0)/close(63), then without any buy rule it should be automatically bought following any sale so long as the # of positions is set to be 1).

Hi Brian,

Why specifically do you say that the result is so different? What is your Annual Return for this strategy on ETFReplay? What differences are you most concerned about?

I was just going to tell you that days and bars are different in P123 and that (generally) there are 20 bars in a month. However, as I am posting this I see that Michael (SpacemanJones) is also advised you regarding this issue at the same time, so I edited this answer. Let us know if this solution resolved your problem.

Chris

Slippage matters. Try this screen I made, which I think is what you want.

Try with 0 slippage vs 0.15 slippage.

https://www.portfolio123.com/app/screen/summary/203581?st=0&mt=9

thank you for all the reply.

I just revised my simulation based on your inputs. However, my results are completely different. It is almost like my ranking system runs in the opposite direction. I am looking to have the ETF, with the highest total returns 13 week. However, the output seems to find the ETF with the lowest total returns 13 weeks. Therefore, the model has chosen SHY (3 month treasury) every month, instead of the ETF with the highest returns.

https://www.portfolio123.com/port_summary.jsp?portid=1521468

the below is the ranking system.

https://www.portfolio123.com/rank_about.jsp

I wonder what I have done wrong. Can you please shed some light on this?

thank you very much.

B

Sorry I forgot to unlock the ranking system. Please see the below the ranking system. I want to find the ETF with the highest 13 week total returns. but the model seems to come up with the etf with the lowest returns. Many thanks

https://www.portfolio123.com/rank_about.jsp?rankid=324563

One thing is that you are using IMW in buy rule …there is no such ETF…is that typo for IWM ?
Check your univ also

thanks, it is IWM, indeed.

I have corrected this.

still not sure what else I have done wrong. thanks

Sell…rule needs modification…once there is investment in SHY…how will it be sold?
Also …if once a particular stock ETF is bought …next period it is only compared with SHY…let’s say it’s not sold. Won’t it miss out on buying the highest ranked etf?

Look at transactions also…and try to find out…if they are as expected…and you will discover issues

Use this as your buy rule:
eval(close(0)/close(60) > close(0,#bench)/close(60,#bench),ticker(“IWM,MDY,QQQ,SPY”),ticker(“SHY”))

And “true” as your sell rule

Allow immediate buybacks and your ranking system should be 13 week price % change

HI PortfolioPerfection

thank you for your comments.

May I know what ‘true’ in the sell rule means? I did make changes as you suggested. the results are here: total returns: 96.76% with a max drawdown of -49% over the period from 31 Oct 2002 to 16 Feb 2018.

https://www.portfolio123.com/port_summary.jsp?portid=1521468

However, according to ETFreplay, the attached are the results: the CAGR is 14.1% with the max drawdown as -18%. I assume no slippage or commissions in Portfolio123 so they re not the reasons.

I attached the screenshot below:

What may be the reason for such a wide difference?


ETFReplay.pdf (234 KB)

I’m joining this discussion because our separate discussion by email seems like duplicated effort.

True as a sell rule means sell/reevaluate at each rebalance. If that’s what you want, then it suggests that you might be better served with a screen (unless you need the transaction record for some reason, like trading with IB).

I would use this as a buy rule:

(Close(0)/Close(63))-1>=(Close(0,GetSeries("SHY"))/Close(63,GetSeries("SHY")))-1

Replace the 63s with however many trading days you’d like. The trick is that SHY will only ever be the top result in your ranking system when there’s nothing else to be bought. RankPos=1 isn’t actually doing anything in portfolio/simulation as long as you’ve set the position size to 100%. (It sounds like you’d like allow immediate buyback set to yes too, but that hammers results in my test.)

I would use

(Close(0)/Close(63))-1

as the node in your ranking system. This is for two reasons: First, it makes the rule above work; there’s no guarantee that Price13W%Chg will give the same result as that formula due to rounding on the back end (as I recall, it’s dividing 251 by 4 to come up with a single quarter, and is only doing so once a week).

In fact, I can guarantee that Pr13W%Chg is going to give a different result than Close(0)/Close(63)-1, because the close formula is taking dividends into account and Pr13W%Chg is only comparing price changes.

If you want dividends to be included, then you should use TotalReturn13W, or change your ranking node to the basic formula. If you don’t want dividends to be included, then use CloseExDiv instead of Close in the basic formula.

Again, I would suggest using Close(0)/Close(63)-1 consistently everywhere.

And the more that I’ve looked at this problem, the less that I think that you’re going to get us to match ETFReplay. I’m not getting anything even remotely as high as they’re reporting.

I have no idea why, but the difference that you’re reporting in drawdowns is probably the key: If they’re reporting an 18% drawdown and we’re reporting a 45% drawdown then we’ve selected different ETFs in 2008/2009.

Whether ETFs pays dividend, whether it is accounted for in P123.

Thanks
Kumar

ETF dividends are paid into the cash account in portfolios and simulations. We adjust the prices for future dividends in screener backtests to ensure total return outputs in that tool.

thank you for your reply.

I did notice that ‘allow immediate buyback’ seriously affects the results negatively.

Do you have any insight as to why “allow immediate buyback” can affect results so much?

the below is the new portfolio.

https://www.portfolio123.com/port_summary.jsp?portid=1521468

In the ETFReplay , the CAGR is 14.1% with -18.4% drawndown (vs 9.8% with -55.2% drawdown of SP500). It outpersorms the SP500 by a large margin. Please see the screenshot in the previous reply.

However in POrtfolio 123, the CAGR is 9.26%, with max drawdown of -26.53%. It underperforms SP500. the annual difference of over 4% between ETFReplay and Portfolio123 is huge.

In the ETFReplay, it picks the best of the four (SPY, QQQ,MDY,IWM) based on the past 3 month returns. If the best is still worse than that of SHY, SHY will be selected instead.

Really wonder which model is doing the right calculation? or maybe I am still not replicating the ETFReplay correctly?

Many thanks for all the support.

B

brwg,

Please, post the transactions for last 5 years, for both ETFReplay and Portfolio 123.
We will compare transaction by transaction manually and figure out where it deviates.

ETFReplay and Portfolio123 are tools, small difference in buy or sell rules make a big difference over period of time.

Thanks
Kumar

thanks for your feedback. Kumar.

I just downloaded the transactions to the attached excel sheet. Yes, I found the major difference is the rebalance timing.

In ETFRepay, rebalance is set at the end of each months to be invested on the 1st business day of the calendar month.

However, in Portfolio123, rebalance frequency is every 4 weeks. Therefore, the rebalance timing is quite different, resulting in buying different ETFs. I think this is the major reason. However, as the difference in returns is quite large, I can’t help wondering whether rebalancing at the end of each calendar month is better than '4 weeks" used in the Portfolio123.

Does anyone know whether Portfolio123 can allow such adjustments for rebalance period?

I am trying to determine:

  1. whether one rebalance timing has advantage over the other
  2. whether there are any other underlying fundamental differences in the two models, potentially causing one model more accurate than the other.

Also, When I set Sell Rule as “True”, what does that mean? Does that mean, stocks will be all sold first before the Buy Rule is applied? If I don’t have “True” in the Sell rule, it seems no stock will be sold at all over the testing period.

Also, why “allow immediate buyback” will affect negatively the results?

Many thanks for your support.


ETFReplay vs Portfolio 123.xls (74.5 KB)

Nice to know, you have nail down the difference.

P123 mostly for fundamental data,
13 weeks quarterly FA report and 52 weeks annual FA report
and weekly rebalance and Monday only trading’s are
one kind of FA strategy relies on earning and weekly fundamentals.

ETFs don’t have any fundamental, it has price and volume, it is more towards technical. TA.

I have followed some TA experts, they have assumed end of the month,mutual funds Money will sell some ETFs for profit and switch another on by first week of every month.

In P123

  1. keep weekly rebalance and
  2. in sell rule Keep
    ((MonthDay>=1) & (MonthDay<=7))
    ============================
    One of the Monday will fall on this dates 1st week of every months, so it become monthly re-balance.

3.In buy rule if you keep
(LastSellDays > 30)
it won’t buy the same ETF just sold for 1 month (30 days) time.


Please, let us know the outcome with transaction history.

Thanks
Kumar :sunglasses:

Kumar

thanks for the reply. I am sorry I am still new to P123.

did you mean “NoBars >=1 and NoBars<=7” in the sell rule?

or “NoDays >=1 and NoDays<=7”?

I tried both and changed the rebalance to weekly. However, I see the trades are done weekly. The rebalance timing is not closer to 1st day of each month and the results not changed much.

Maybe I have misunderstood you?

thanks

sorry, I will try again. I seem to have misunderstood you. Let me try again.