I’m sure this will be a waste of my time, but I am seeing noticeable differences in backtested performance of my primary strategies between the Beta website and the Current production version.
I would not have expected this to be the case, since I assumed the Beta calculation engine just became the Current production engine.
Is there a list of the differences between the Current production engine and what was running on the Beta site?
*The following post set my expectation for what we should be seeing now and which does not seem to be the case at all:
*For users that want functions to operate on ‘bars’ (trading days)
You don’t need to do anything (almost). This is for users that want to design strategies in single country universes where all stocks in the universe stop trading on the same days. All technical functions operate on bars by default. So for example Close(10) is the close 10 trading days ago (typically 2 week span unless there was a holiday)
There’s only one case where you need to intervene: if you use sector and industry series. We chose to create industry and sector time series for the entire North America which meant that the time series had to be based on weekdays and include holidays.
This can cause problems in your strategy if, for example, you like looking for stocks that have outperformed the industry. For that filter to work the period has to be in the same units. You will need to rewrite your formulas using weekday functions. For example if you are looking for stocks that have outperformed the industry like this:
*I am confused, for example, by what dailyavgtot(30) now represents. Is it the average daily dollar volume for the last 30 bars? Is it the average daily volume for weekdays during the last 30 calendar days with double values given for days before holidays?
Hugh, I think you might have accidentally posted to the wrong thread. Although, glad to know I am not the only one still having problems with the recent changes.
Daniel, Yield is you main factor. This was also released in production that was not on the beta site. It is a workaround we added that “fixes” yield for many ADRs (but not all)
That’s fair. Is that the only difference between the Current Production Server and the Beta Site. I am seeing differences (albeit not as large) in other strategies that do not rely on Yield.
The annualized return only decreased by ~100bps, but the returns in individual years are quite different.
The Ranking System for this model has many nodes, so it doesn’t seem reasonable to ask you to figure out what might have changed for all of them. I was hoping that you could just provide me a change log from 4/30 on the Beta site to the Current Production Version, that way I can try to understand what has changed. I find that I typically learn some things when I go through one of these exercises. I am actively trying to replace this model, so the learnings would be helpful.
The other differences I was referring to might have already been in place in beta. I have to verify that. So at this point Yield is the only one I can blame.
NOTE: The differences I’m talking about with old production have to do with technicals like AvgDailyTot, which you use. The old production was using a dividend unadjusted price series , now we only adjust for future dividends at the end of the calculation. The new version is slightly less “correct” if during your period , say 60 bars, there was a dividend. Since we only adjust once at the end using the latest future dividend adjustment for the observation date, the past dividend in the 60 bars is not accounted for. Difference should be insignificant . We did this change because keeping around a dividend unadjusted series for very few use-cases was becoming too expensive.
I tested my models, annual return decline by 1 ~ 3% for all models. There are also big differences(3% ~ 15%) for some year’s return between old prod and new prod. I’m only using bar based functions for only US stocks.
As per the design, it shall not be impacted unless there are still other changes hidden. Don’t understand why.
We found a bug that MIGHT be causing the differences in Daniel’s systems. It’s an extremely minor bug, affecting just a handful of stocks. We’ll roll out the fix with the nightly update.
Yuval, the bug you identified seems to have been the source of the problem. My backtests are matching the performance I was getting on the beta server (except for my high yield strategy which has been effected by the ADR Yield calculation change).
Thanks Yuval. I did some test yesterday looks normal even till just 1 hour ago. When i was re-running one of my existing strategies around 23:30PM EST, the annual performance of the strategy dropped around 6%suddenly, i tried other strategies and got similar results - all dropped around 6~8%.
Would like to know if there is any new changes deployed to production around that time period? Many thanks
hi P123 staffs/Yuval, I believe more and more people will encounter this issue. You can also run some of your existing model to see. If you did have some new changes deployed during past hours, can we roll back to the previous version before Monday and deploy the new version to beta site so we can figure out which factor/formula causing the issue?
Same problem for me. I just rerun some SIm (without changing period) and almost all dropped drammatically. with different pattern. I am not speaking of beta site but normal site.