Difference Between the Beta Site and the Current Production Version

Research
1

I’m sure this will be a waste of my time, but I am seeing noticeable differences in backtested performance of my primary strategies between the Beta website and the Current production version.

I would not have expected this to be the case, since I assumed the Beta calculation engine just became the Current production engine.

Is there a list of the differences between the Current production engine and what was running on the Beta site?

Thanks,

Daniel

2

For example, this strategy was run on the Beta Site on May 7th and has a TTM return of ~30%.

https://www.portfolio123.com/port_summary.jsp?portid=1640627

This number seems reliable as it closely matches the TTM performance of Live version of the same model.

https://www.portfolio123.com/port_summary.jsp?portid=1514728

However, I reran the backtest today on the Current Production Server over the same time period and it has a TTM return of ~10%.

https://www.portfolio123.com/port_summary.jsp?portid=1681183

3

*The following post set my expectation for what we should be seeing now and which does not seem to be the case at all:

*For users that want functions to operate on ‘bars’ (trading days)

You don’t need to do anything (almost). This is for users that want to design strategies in single country universes where all stocks in the universe stop trading on the same days. All technical functions operate on bars by default. So for example Close(10) is the close 10 trading days ago (typically 2 week span unless there was a holiday)

There’s only one case where you need to intervene: if you use sector and industry series. We chose to create industry and sector time series for the entire North America which meant that the time series had to be based on weekdays and include holidays.

This can cause problems in your strategy if, for example, you like looking for stocks that have outperformed the industry. For that filter to work the period has to be in the same units. You will need to rewrite your formulas using weekday functions. For example if you are looking for stocks that have outperformed the industry like this:

Close(0)/Close(10) > Close(0,#industry)/Close(20,#industry)

You should rewrite it using weekday functions since the industry series includes holidays:

Close_D(0)/Close_D(10) > Close_D(0,#industry)/Close_D(20,#industry)

*I am confused, for example, by what dailyavgtot(30) now represents. Is it the average daily dollar volume for the last 30 bars? Is it the average daily volume for weekdays during the last 30 calendar days with double values given for days before holidays?

Thank you!

Hugh

4

Hugh, I think you might have accidentally posted to the wrong thread. Although, glad to know I am not the only one still having problems with the recent changes.

Cheers,

Daniel

5

Daniel, Yield is you main factor. This was also released in production that was not on the beta site. It is a workaround we added that “fixes” yield for many ADRs (but not all)

https://www.portfolio123.com/mvnforum/viewthread_thread,13320

6

That’s fair. Is that the only difference between the Current Production Server and the Beta Site. I am seeing differences (albeit not as large) in other strategies that do not rely on Yield.

7

I believe there are other ratios that differ in minor ways. I would need to see the strategy.

8

Thanks Marco. This strategy also seems to have changed more than I would have expected.

Run On Beta Site on 4/30/2022
https://www.portfolio123.com/port_summary.jsp?portid=1680443

Run on Current Prod Version 5/13/2022
https://www.portfolio123.com/port_summary.jsp?portid=1681266

The annualized return only decreased by ~100bps, but the returns in individual years are quite different.

The Ranking System for this model has many nodes, so it doesn’t seem reasonable to ask you to figure out what might have changed for all of them. I was hoping that you could just provide me a change log from 4/30 on the Beta site to the Current Production Version, that way I can try to understand what has changed. I find that I typically learn some things when I go through one of these exercises. I am actively trying to replace this model, so the learnings would be helpful.

Thanks,

Daniel

9

Yield is still quite relevant in those systems.

The other differences I was referring to might have already been in place in beta. I have to verify that. So at this point Yield is the only one I can blame.

NOTE: The differences I’m talking about with old production have to do with technicals like AvgDailyTot, which you use. The old production was using a dividend unadjusted price series , now we only adjust for future dividends at the end of the calculation. The new version is slightly less “correct” if during your period , say 60 bars, there was a dividend. Since we only adjust once at the end using the latest future dividend adjustment for the observation date, the past dividend in the 60 bars is not accounted for. Difference should be insignificant . We did this change because keeping around a dividend unadjusted series for very few use-cases was becoming too expensive.

10

Were there other changes to Yield besides the ADR specific change? The models linked to above have ADRs excluded from the universe.

Did the handling of AvgDailyTot change from the beta server to the current production server?

11

I tested my models, annual return decline by 1 ~ 3% for all models. There are also big differences(3% ~ 15%) for some year’s return between old prod and new prod. I’m only using bar based functions for only US stocks.
As per the design, it shall not be impacted unless there are still other changes hidden. Don’t understand why.

12

We found a bug that MIGHT be causing the differences in Daniel’s systems. It’s an extremely minor bug, affecting just a handful of stocks. We’ll roll out the fix with the nightly update.

13

Thanks Yuval

14

Yuval, the bug you identified seems to have been the source of the problem. My backtests are matching the performance I was getting on the beta server (except for my high yield strategy which has been effected by the ADR Yield calculation change).

15

Thanks Yuval. I did some test yesterday looks normal even till just 1 hour ago. When i was re-running one of my existing strategies around 23:30PM EST, the annual performance of the strategy dropped around 6%suddenly, i tried other strategies and got similar results - all dropped around 6~8%.
Would like to know if there is any new changes deployed to production around that time period? Many thanks

16

I also just got some very large changes in my model results - has anything changed?

17

hi P123 staffs/Yuval, I believe more and more people will encounter this issue. You can also run some of your existing model to see. If you did have some new changes deployed during past hours, can we roll back to the previous version before Monday and deploy the new version to beta site so we can figure out which factor/formula causing the issue?

Thanks.

18

I am seeing big changes also, 1st sim I checked had AR drop from 32% to 22%…

19

Same problem for me. I just rerun some SIm (without changing period) and almost all dropped drammatically. with different pattern. I am not speaking of beta site but normal site.

20

I am seeing the same problem - however it seems not be the same for all models. Some have dropped around 6 to 8%, while a few have increase 6 to 8%.