Index | Recent Threads | Who's Online | Search

Posts: 373    Pages: 38    Prev 2 3 4 5 6 7 8 9 10 11 Next
Last Post
This topic has been viewed 18550 times and has 372 replies
atw
Re: FactSet beta site v1.0, NOW LIVE

Hi Yuval and Aaron,

I have begun to dissect and rebuild my models one input at a time.

Although it probably goes without saying, I am focusing all of my attention right now on understanding differences between the old CompuStat using preliminary data vs. the new Beta CompuStat w/preliminary data option.

To make things easy, I thought, I would start with something simple: COUNTRY(“XXX”)=FALSE

I don’t believe the COUNTRY function is working on the Beta site.

The good and bad news about this discovery, if confirmed, is that I started by investigating just one p123 function and it does not appear to be working properly on the Beta site.

COUNTRY is not that important a function in and of itself but perhaps it will point to other malfunctions that cumulatively narrow the differences between existing and the Beta site results?

Thank you.

Hugh

Apr 21, 2020 3:18:09 PM       
aschiff
Re: FactSet beta site v1.0, NOW LIVE

Hugh,
In my previous post, I was talking exclusively about changes to the beta server, not the production server.

1) The code for preliminary fallbacks has been refactored, but the main difference should be the segregation of annual preliminaries. If you find a specific factor or function that's noticeably different between production and beta (Compustat & Use Prelim), we can further discuss that in this thread.
2) Regarding yesterday's fixes, if you're still seeing drastic differences, we'll need to look further into what's causing the change.
3) Intentional changes have not been formally tracked, but they have been mentioned in this thread.
4) None of the changes here have been put into production yet; it's going to remain in beta until we're satisfied.

Having said all that, rest assured that we're not aware of anything being catastrophically broken in production.

I will create an issue for Country and will look into it shortly. Thanks for finding it.

Aaron
Portfolio123 Staff

Apr 21, 2020 3:39:27 PM       
aschiff
Re: FactSet beta site v1.0, NOW LIVE

Marco, this simple sim does not work on the new platform, with Compustat or Factset data.

Buy: Eval( Ticker("SPY") & Portfolio(1451475),ticker("SPY"),ticker("ief"))
Sell: nodays>12

Attached are sims on current and new platform.


Georg, this issue has little to do with the beta server. I have already answered regarding the 'undefined' behavior of systems defined like this in your Different results for same Buy rule thread. Consider the scenario where 1451475 is holding SPY. In that case, your rule would evaluate to true for both SPY and IEF, so it would end up choosing the best ranked ETF of the two, which in your case is undefined. Due to the change of an internal structure, what you're actually seeing is a more stable 'undefined' rank. You should be using FSum or FCount to ensure that your market-timing condition is the same for all ETFs evaluated.

Aaron
Portfolio123 Staff

Apr 21, 2020 4:57:05 PM       
Edit 1 times, last edit by aschiff at Apr 21, 2020 4:58:09 PM
atw
Re: FactSet beta site v1.0, NOW LIVE

Hi Aaron,

Thank you for your post.

Could you please choose several companies and affected variables, and then present before/after numbers which have changed because of the corrected method by which p123 is handling preliminary data on the Beta site?

Also, is p123 planning to make this change to the production site? If so, when?

If this is the only change p123 intends to make to the production site it may be easier to determine if this change involving preliminary data is causing the differences I’m seeing in Beta site results. If p123 is not planning to make changes to the production site, or if the changes include more items than just the preliminary data – not sure this is feasible but… - if p123 could create a third site which would essentially be the production site with only changes to the preliminary data it would be easy to determine the extent to which this (vs. this plus additional issues that may exist like COUNTRY) are the source of the differences.

I looked at UNIVERSE after looking at COUNTRY. I believe this function is working correctly on the Beta site but it’s hard from the outside to be 100% sure because it’s impossible to feel certain about my “control” on the Beta site given other possible issues.

Finally, I think it should be mandatory for p123 to track (and even publish) its changes. P123 has a dedicated group of highly interested and sympathetic users. Many, as demonstrated in the Forum, are very experienced and expert across a wide range of disciplines. Everybody understands it’s easy for small hidden logic errors to create nightmares when programming. Why not provide the detail so that p123’s subscribers can get more involved? Testing one input at a time, as I’ve started to do… It’s really hard to track down these issues from the outside.

Hugh

Apr 21, 2020 5:04:14 PM       
dnevin123
Re: FactSet beta site v1.0, NOW LIVE

Differences in ROI% Calculation example:

For Ticker("DHT") if I look at the calculated ROI% on the Production Server and on the Beta Server using Prelim I see fairly big differences in the calculated values.


Production Server
ROI%5YAvg = 3.88
ROI%TTM = 6.07
ROI%PTM = -0.66

Beta Server
ROI%5YAvg = 5.10
ROI%TTM = 6.61
ROI%PTM = -0.32

Is this due to the way preliminary data is being handled? Seems odd that it would have a larger effect on the 5YAvg then on the TTM and PTM calculations.

Thanks,

Daniel

Apr 22, 2020 10:21:18 AM       
Edit 1 times, last edit by dnevin123 at Apr 22, 2020 10:24:10 AM
dodge1664
Re: FactSet beta site v1.0, NOW LIVE

When running one of my sims, I get

Ranking failed on 01/02/1999: Invalid criteria in Rule 6. Error near 'FINANC': Invalid command 'FINANC'

rule 6 of my Universe contains:
Sector!= FINANC

Apr 22, 2020 10:32:35 AM       
dnevin123
Re: FactSet beta site v1.0, NOW LIVE

I think this might have been reported already, but I am getting NA for WeeksToY for all of the securities I have examined.

Apr 22, 2020 11:09:26 AM       
Chipper6
Re: FactSet beta site v1.0, NOW LIVE

Differences in ROI% Calculation example:

For Ticker("DHT") if I look at the calculated ROI% on the Production Server and on the Beta Server using Prelim I see fairly big differences in the calculated values.


Production Server
ROI%5YAvg = 3.88
ROI%TTM = 6.07
ROI%PTM = -0.66

Beta Server
ROI%5YAvg = 5.10
ROI%TTM = 6.61
ROI%PTM = -0.32

Is this due to the way preliminary data is being handled? Seems odd that it would have a larger effect on the 5YAvg then on the TTM and PTM calculations.

Thanks,

Daniel


Daniel, you might want to try comparing the individual line items that comprise ROI. See the docs. That should give you a better idea.

Apr 22, 2020 11:11:58 AM       
geov
Re: FactSet beta site v1.0, NOW LIVE

Marco, this simple sim does not work on the new platform, with Compustat or Factset data.

Buy: Eval( Ticker("SPY") & Portfolio(1451475),ticker("SPY"),ticker("ief"))
Sell: nodays>12

Attached are sims on current and new platform.


Georg, this issue has little to do with the beta server. I have already answered regarding the 'undefined' behavior of systems defined like this in your Different results for same Buy rule thread. Consider the scenario where 1451475 is holding SPY. In that case, your rule would evaluate to true for both SPY and IEF, so it would end up choosing the best ranked ETF of the two, which in your case is undefined. Due to the change of an internal structure, what you're actually seeing is a more stable 'undefined' rank. You should be using FSum or FCount to ensure that your market-timing condition is the same for all ETFs evaluated.

Aaron,
Eliminating ticker IEF and using VTI instead of SPY, and using FSum, then buy rule becomes:
Eval(FSum("$port1",#All)>0,ticker("VTI"),ticker("0")) // $port1=Ticker("SPY")&Portfolio(1451475)

This would only evaluate to true when port 1451475 is holding SPY. So it can only select VTI when true and no ranking is involved.
The backtest from 2000 produces AR= 12.33% and Overall Winners= (36/41) 87.80% on existing P123 platform.

On the new platform with Compustat with prelim:
1st run: AR= 11.46% and Overall Winners= (24/27) 88.89%
2nd run: AR= 5.62% and Overall Winners= (1/1) 100.00%

On the new platform with FactSet with prelim:
1st run: AR= 10.31% and Overall Winners= (10/13) 76.92%
2nd run: AR= 5.65% and Overall Winners= (2/2) 100.00%

So new platform is not stable for rules with "Portfolio" function. That's all one can conclude.

Test Timer (VTI-cash)
https://www.portfolio123.com/port_summary.jsp?portid=1606919

Here is another buy rule which is not affected by ranking (VTI-IEF):
FSum("1*$port1",#All)>0 & ticker("vti") | FSum("1*$port1",#All)<1 & ticker("ief") // $port1=Ticker("SPY")&Portfolio(1451475)

The backtest from 2000 produces AR= 17.76% and Overall Winners= (64/82) 78.05% on existing P123 platform.

On the new platform with Compustat with prelim:
1st run: AR= 5.38% and Overall Winners= (6/9) 66.67%
2nd run: different to above, 3rd run is different again, and so on.

On the new platform with FactSet with prelim:
1st run: AR= 5.62% and Overall Winners= (1/1) 100.00%
2nd run: AR= 6.26% and Overall Winners= (3/3) 100.00%

Apr 22, 2020 1:17:28 PM       
Edit 2 times, last edit by geov at Apr 23, 2020 10:41:57 AM
yuvaltaylor
Re: FactSet beta site v1.0, NOW LIVE

We've found some issues with the way a number of annual items were being handled and have put in a fix for those. Specifically, annuals were being pulled from interim statements. This impacted, among others, NoIncP4YN2Y, EPSEst, EPSActual, EPSSurprise, SalesEst, SalesActual, SalesSurprise, QtrComplete, DivPS5YAvg, and PeriodDateA. I believe this is now fixed.

See the Trello board for the status of the other bugs and fixes, and thanks for all your reporting.

Yuval Taylor
Product Manager, Portfolio123
invest(igations)
Any opinions or recommendations in this message are not opinions or recommendations of Portfolio123 Securities LLC.

Apr 22, 2020 3:00:27 PM       
Posts: 373    Pages: 38    Prev 2 3 4 5 6 7 8 9 10 11 Next
 Last Post