New user here.
I am running through the tutorials on simulation.
The tutorial sim uses a data range of: 01/02/07 → 10/10/17
I used the same data range as I followed along, with the same rules and settings.
But my results are different.
The sim had a total return of 240% and my sim had a total return of 205%.
All the transactions look very different as well. Its possible I mistyped a rule or settings somewhere but I don’t think so.
My question is this. Should two sims, using the same data, but performed years apart, give the same results?
Thanks
Tony
Those tutorials were based on Compustat data, and now we’re using FactSet data, so the results will definitely be different.
Ok thanks. Does that imply that one or both datasets is inaccurate in some way?
How long ago was that change made?
Thanks again.
Tony
Even with prior data historicals would be corrected/updated by compustat if errors were found, which could change results. Depending on the sim one small change in result can cascade into larger downstream results.
Change to Factset happened sometime this year. fwiw, the performance results of my models (mostly kitchen sink, multifactor type) have been similar to what I saw with compustat. Some others have reported more significant differences - likely due to how different companies might interpret and standardize the data to conform to a database format. There are some threads on here about it somewhere.
Quick question to Yuval: Is the database “PIT Method - Use” is more likely to have past data (and thus ranks) updated/adjusted than the “Exclude” database?
No, both datasets are very accurate. We had to make the change because Compustat insisted that many of our users be licensed through them, which would have cost those users thousands of dollars per year, and they wanted to shorten the lookback period for non-licensed users.
The change was made at the end of June 2020.
Compustat and FactSet differ quite a bit in the way they calculate certain factors, and we’ve done our best to smooth out those differences. Another large difference is that FactSet offers a lot more stocks in the All Fundamentals database (particularly ADRs). The way FactSet handles preliminary data is different from how Compustat handled that data. And with FactSet, for pre-March 2020 filings, we no longer have “effective dates,” i.e. the date on which the filing was processed by the data provider.
If you would prefer to use Compustat data, you can still do so with a license from them.