I have been watching the historical performance of a very simple portfolio over the last several days. Each time I run it I record a different performance. The equity curve recorded over several different days has a correlation of 92% , one would expect it to be exactly 100%. Over 20 years the cumulative performance can vary by 50% every time you run it! This problem seems to have emerged since the change to factset. Very worried to see such inconsistencies.
InconsistentPortfolio.xlsx (373 KB)
FactSet revises old data when it finds inconsistencies or mistakes; in addition, there are plenty of restatements. Compustat did this to a small degree, but FactSet does it much more. I think you should stop expecting absolute consistency in your backtests.
The changes are more prominent in the recent history , last several days etc. keep changing on a daily basis . The recommended portfolio also seems to change abruptly and significantly. So I am more worried about the trading going forward as oppose to the back test. Do you still provide compusstat feed as an alternative.
Thank you for your responses
A critical issue was fixed around this weekend that could explain the extreme changes you noticed. We’re also watching for changes as we move forward and will continue to address additional issues discovered thereby.
The performance inconsistency is really worrying. On 8th July 2020 I saved the back test of a strategy. On 20 July I recomputed the back test. The correlation is 11% . If I re run it tomorrow , I am pretty sure i will get a totally different number. The constituents on 8th Jul are way different to the constituents on 20 th July run .
I don’t know what to say to this. A lot depends on what sort of backtest you’re running. If I run a simulation that I ran a month ago and the number of stocks bought is relatively small in comparison to the universe, I’m going to expect that about 5% of my buys and sells will be different. The longer the period, the bigger the difference that will make. The most useful piece of advice I can give you is to download the transaction record and compare the actual buys and sells. You’ll see that often a stock that ranked #18 is now ranked #16. If you’re buying the top 20 stocks, that means that a company that ranked #19 (and was bought) might now rank #21 (and will now be skipped).
If you want to share the precise system that’s giving you trouble, I’d be happy to take a closer look.
Thank you , I have attached a file with all details.
InconsistentPortfolio2.xlsx (678 KB)
pf123_help.xlsx (15.1 MB)
please use the file named “pf123_help”
Mike -
How many stocks are specified in the “rebalance” tab?
Your buy rules will allow around 250 stocks to pass. Let’s say the number of stocks specified in the rebalance tab is 20. So every week, your simulation will choose the 20 stocks ranked closest to 10, e.g. with ranks 9.68 to 9.99. That will be a very unstable group of stocks. In terms of ranking, I’d be hard-pressed to come up with a more unstable group. Even one small revision in one stock could cause a real overhaul.
Look at your transaction record. You’ll see the ranking of every stock you buy or sell.
Ranks are far more stable at the high end and low end of a ranking system–close to 100 and close to 0. They’re even more stable if you put the buy rules in a custom universe. Ranking All Fundamentals and then choosing those that are closest to 10 is going to give you a very unstable system indeed.
I have specified 20 in the rebalance tab.
What do you mean by “They’re even more stable if you put the buy rules in a custom universe. Ranking All Fundamentals and then choosing those that are closest to 10 is going to give you a very unstable system indeed” . I have chosen my universe to be ALL Fundamental USA. Is there a bigger universe? You suggest taking a smaller universe?
So your point is revisions in data is causing the performance to change. I can understand that to be the case for recent dates. Are you saying factset is revising data 20 years ago? Please look at column N of the sheet "pnl backtest " in the workbook pf123_help I sent you. I have taken the day on day returns. It i sthis column that changes for dates 10-15 years ago. Recent revision should not be affecting data 15 years ago ?
So you’re ranking 8,500 stocks and each week you’re choosing the twenty stocks that rank closest to 10. If you didn’t have any buy rules, you’d be choosing the stocks that rank between 9.76 and 9.99. Let’s say your buy rules eliminate half of those. So now you’re choosing the stocks that rank between 9.53 and 9.99. You’re ranking on six different factors. Now FactSet goes back and changes the PE of a stock ten years ago because that’s what they do–they often update things where they find a mistake or where there’s a restatement. Or maybe ICE data discovers a mistake in a split and changes the price of a stock. It doesn’t matter if it’s one of the stocks that you chose–it could be any stock. So now a stock that ranked 9.54 is ranked 9.52. It doesn’t make the cut and another stock makes the cut.
Now if you put your buy rules into the universe instead, you’d be ranking 4,250 stocks instead of 8,500, and you’d be choosing stocks that rank between 9.53 and 9.99 BUT those ranks would be more stable. Why? Because the big changes will be more likely to affect tiny stocks than large ones and because changes to what is ranked precisely 10 and not 10.01 or 9.99 will be fewer. Consider this. With 8,500 stocks in the universe, 850 are going to be ranked under 10 each week, of which half will pass your buy rules. That population will have a turnover rate of, say, 10% a week. Because you’re choosing 20 out of those stocks, your turnover rate is going to be huge. With 4,250 stocks ranked per week, the turnover rate will be lower, and you’ll be choosing a smaller percentage of the 425 stocks. Try it–you’ll see that your turnover will go down.
If you want a really stable simulation, I suggest putting your buy rules in your universe rules, reversing your ranking system, and taking the top 20 stocks. Your turnover will go way down.