Index | Recent Threads | Who's Online | Search

Posts: 373    Pages: 38    Prev 1 2 3 4 5 6 7 8 9 10 Next
Last Post
This topic has been viewed 18556 times and has 372 replies
Chipper6
Re: FactSet beta site v1.0, NOW LIVE

If it was up to me, I'd focus first on exposing all the data before trying to extinguish the bugs. That's how it was done last time during the switch from Reuters and it was an amazing job.

Apr 20, 2020 3:00:01 PM       
yuvaltaylor
Re: FactSet beta site v1.0, NOW LIVE

Hi Yuval,

Thank you very much for your update. Let me make sure I understand what you are saying.

1) Is there a difference in meaning between preliminary data and interim data?

Yes. Interim data is quarterly data. Quarters are interim periods between annual data releases. Preliminary data is released by the company prior to the SEC filing.

2) Does p123's error mean that preliminary annual data was backdated in p123 so that simulations and backtests believed this data was available up to one year before it really was?

No. When a statement is flagged as preliminary, some NAs will be filled by going back to the prior period. Since the annual was flagged incorrectly as prelim, NA is the correct value, and falling back could get you a false positive.

3) Are these backdating errors limited to annual CompuStat data only or do they affect such p123 functions as "ttm" and/or other non-CompuStat data such as estimates, short interest or institutional ownership?

These errors are limited to Compustat fundamentals only and do not affect estimates, short interest, institutional ownership, etc. Those are all completely different datasets.

Yuval Taylor
Product Manager, Portfolio123
invest(igations)
Any opinions or recommendations in this message are not opinions or recommendations of Portfolio123 Securities LLC.

Apr 20, 2020 10:11:10 PM       
yuvaltaylor
Re: FactSet beta site v1.0, NOW LIVE

By the way, a few of the bugs have now been fixed. Not all of them, but issues relating to preliminaries and invalid benchmark notices should be OK now.

Yuval Taylor
Product Manager, Portfolio123
invest(igations)
Any opinions or recommendations in this message are not opinions or recommendations of Portfolio123 Securities LLC.

Apr 20, 2020 11:20:28 PM       
Hedgehog
smile Re: FactSet beta site v1.0, NOW LIVE

Possible to share some kind of bugtracking with the community in order to improve the bugreporting in this thread? E.g. by editing the first post with something like this:

#001: The quick brown fox jumps over the lazy dog.
STATUS: Closed
#002: The five boxing wizards jump quickly.
STATUS: Open
etc.

Apr 21, 2020 11:01:55 AM       
atw
Re: FactSet beta site v1.0, NOW LIVE

Hi Yuval,

Thank you for your detailed explanation.

It sounds as if interim, meaning quarterly, preliminary data has always been handled correctly. Is that so?

If quarterly preliminary data has always been handled correctly, then “ttm” data points should be unaffected too. Is that also so?

I wonder if you could provide some concrete examples, meaning before the correction and after, of how annual data points may have changed?

I ask because I’m not hearing enough of a cataclysmic difference regarding what p123 has corrected to explain the huge differences in simulation results I am seeing.

Therefore, either…

1) I am misunderstanding, and underestimating, just how significant this correction to the annual data must be.

2) There are other changes to or bugs affecting the beta site w/preliminary data that need attention as well.

Unfortunately, I don’t believe it’s possible to evaluate the transition to the FactSet data without fully understanding whatever issues are affecting the CompuStat w/preliminary data – old or new.

In addition, and in part subject to whether “ttm” data is affected, I’m not sure I even use that many annual data points in my screens and models. If I (and hopefully other users) can understand exactly which data points are at issue, it will be much easier to correct, discard or recalibrate our existing models.

At that point we can turn our attention to evaluating the FactSet data with more certainty - and while there is still time.

I understand p123 is striving to get things right. I appreciate your help with this.

Hugh

Apr 21, 2020 12:07:21 PM       
aschiff
Re: FactSet beta site v1.0, NOW LIVE

To reiterate what Yuval said, the previous version of the beta server (before 2020-04-20 6PM CDT) was rife with mistakes relating to handling preliminary fallbacks. The latest beta version will more closely match current behavior in production for handling preliminary data, but it will not match exactly since intentional changes are still present. It is recommended that you run backtests again that were wildly different before the fixes.

To give a sense of the magnitude if the issues, here is a list of the affected factors and functions I could find. These issues also impacted any industry factors derived from them.

Standalone factors: DbtS2NIQ DbtS2NITTM EBITDMgn%5YAvg EBITDMgn%TTM EBITDPSTTM EPS#Positive EPS%ChgA EPS%ChgPQ EPS%ChgPYQ EPS%ChgTTM EPS3YCGr% EPS5YCGr% EPS10YCGr% EPSPExclXorA EPSPExclXorQ EPSPExclXorTTM EPSPInclXorA EPSPInclXorQ EPSPInclXorTTM EPSStableQ GMgn%5YAvg IncPerEmpQ IncPerEmpTTM IntCovQ IntCovTTM NI%ChgPQ NI%ChgPYQ NI3YCGr% NI5YCGr% NoIncP4YN2Y NoPosEBITDA5Y NoPosEPS5Q OCFPSQ OCFPSTTM PayRatio5YAvg PEExclXorTTM PEG PEGY PEInclXorTTM PMgn5YCGr% Pr2SalesQ Pr2SalesTTM Prc2SalesIncDebt Retn%Q Retn%TTM ROA%5YAvg ROE%5YAvg ROI%5YAvg Sales%ChgA Sales%ChgPQ Sales%ChgPYQ Sales%ChgTTM Sales3YCGr% Sales5YCGr% Sales10YCGr% SalesPerEmpQ SalesPerEmpTTM SGA2SalesQ SGA2SalesTTM
Functions (including their A, Q, and TTM factors): AstTurn CapEx2Sales Cash2AstTot CurRatio DbtLT2Ast DbtLT2Cap DbtLT2Eq DbtTot2Ast DbtTot2Cap DbtTot2Eq EBITDAMgn% FCFLMgn InvTurn OPMgn% PayRatio PEGInclXor PTMgn% QuickRatio RecTurn ROA% ROE% ROI% SGA2Sales%

Aaron
Portfolio123 Staff

Apr 21, 2020 12:46:03 PM       
Edit 2 times, last edit by aschiff at Apr 21, 2020 4:17:56 PM
yuvaltaylor
Re: FactSet beta site v1.0, NOW LIVE

Possible to share some kind of bugtracking with the community in order to improve the bugreporting in this thread? E.g. by editing the first post with something like this:

#001: The quick brown fox jumps over the lazy dog.
STATUS: Closed
#002: The five boxing wizards jump quickly.
STATUS: Open
etc.


We thought this was a very good suggestion so we have a bug-tracking site now open for visits. See it here:

https://trello.com/b/6nucgADo/factset-bugs

If you see bugs that are not on this site, please comment below.

Yuval Taylor
Product Manager, Portfolio123
invest(igations)
Any opinions or recommendations in this message are not opinions or recommendations of Portfolio123 Securities LLC.

Apr 21, 2020 1:26:37 PM       
Edit 1 times, last edit by yuvaltaylor at Apr 21, 2020 1:38:28 PM
yuvaltaylor
Re: FactSet beta site v1.0, NOW LIVE

Hi Yuval,

Thank you for your detailed explanation.

It sounds as if interim, meaning quarterly, preliminary data has always been handled correctly. Is that so?

If quarterly preliminary data has always been handled correctly, then “ttm” data points should be unaffected too. Is that also so?

I wonder if you could provide some concrete examples, meaning before the correction and after, of how annual data points may have changed?

I ask because I’m not hearing enough of a cataclysmic difference regarding what p123 has corrected to explain the huge differences in simulation results I am seeing.

Therefore, either…

1) I am misunderstanding, and underestimating, just how significant this correction to the annual data must be.

2) There are other changes to or bugs affecting the beta site w/preliminary data that need attention as well.

Unfortunately, I don’t believe it’s possible to evaluate the transition to the FactSet data without fully understanding whatever issues are affecting the CompuStat w/preliminary data – old or new.

In addition, and in part subject to whether “ttm” data is affected, I’m not sure I even use that many annual data points in my screens and models. If I (and hopefully other users) can understand exactly which data points are at issue, it will be much easier to correct, discard or recalibrate our existing models.

At that point we can turn our attention to evaluating the FactSet data with more certainty - and while there is still time.

I understand p123 is striving to get things right. I appreciate your help with this.

Hugh

Hugh,

Now that some of the bugs have been fixed, please report specific differences you're seeing between the regular site and the beta Compustat Use Prelims site. Try to isolate exactly what the differences are. For example, if you're running a simulation on both sites, look at the transactions record; if you're doing a screen, use the ShowVar and Ticker commands to see what the differences might be due to; if you have differences in rank performance, try to figure out which node is causing it. All of these will help us sort out any remaining bugs in the site. After all the bugs have been fixed, the beta site will not be identical to the regular site because of the difference in how preliminaries are handled, but the results should not vary too widely.

Also see Aaron's comment above and the list of bugs that have not yet been resolved on the Trello board I posted.

Yuval Taylor
Product Manager, Portfolio123
invest(igations)
Any opinions or recommendations in this message are not opinions or recommendations of Portfolio123 Securities LLC.

Apr 21, 2020 1:37:46 PM       
rodbarcv2
Re: FactSet beta site v1.0, NOW LIVE

A few of my sims work now, and all of them have been consistent to have BETTER results (higher Sharpe ratio, better performance) with FactSet + Exclude Prelim than with Compustat (with or without Prelim).

However, many sims are still unable to run, all of them with the same error: Invalid uid in PriceHistory

One of the sim with error for reference: https://beta.portfolio123.com/port_summary.jsp?portid=1531521

Apr 21, 2020 1:46:38 PM       
Edit 1 times, last edit by rodbarcv2 at Apr 21, 2020 1:47:41 PM
atw
Re: FactSet beta site v1.0, NOW LIVE

Dear Aaron,

Thank you for your post.

1) Clearly these revisions involve a LOT more than just annual datapoint corrections.

2) Running the sims post 2020-04-20 at 6PM CDT made virtually no difference to the results I am seeing.

3) Have all of the intentional changes been posted on the bug-tracking site?

4) Sim results are apparently unchanged on the old site.

The consequences of what you are saying are horrifying. It appears that every serious fundamentals-based formula, ranking system, screen, simulation and/or designer model that relied upon CompuStat data populating p123 before yesterday evening was constructed using a significant amount of data unavailable at the time the data was purported to have been available.

Am I wrong?

Hugh

Apr 21, 2020 1:52:52 PM       
Posts: 373    Pages: 38    Prev 1 2 3 4 5 6 7 8 9 10 Next
 Last Post