Index | Recent Threads | Who's Online | Search

Posts: 17    Pages: 2    1 2 Next
Last Post
New Thread
This topic has been viewed 758 times and has 16 replies
Monday33
Open question about different simulation platforms

Hi all!!

I would like to comment something with you and get some feedback because I'm not sure about what to think.

Recently I tried another plattaform, very similar to Portofio 123, but little bit simplier. The staff from the plattaform send some systems to the users. One I recibed shows a very good performance, around 30% a year, then I translated that system to the Portfolio123 just to check if the results are the same, and... surprise, the sim goes to ruin in just few years.

*I'm not going to write down here the name of that platform, if anyone is interested ask me by private email and I will send you the adress.

I asked to the guys who run the plattaform about that difference in results. And they answer the following:

“The key difference I found where the two systems are different is that P123 generally takes a standard universe of all stocks, ranks it and then applies your buy rules, whereas "X" applies the universe rules, then ranks, then applies any buy rules. This is a much bigger difference than it sounds, as ranks act very differently with different numbers of securities in a universe. If you look at your Value strategy and click the Rank Screener option you can see that it includes just 922 securities in your list, whereas P123 will be ranking on approximately 4,500. Even a difference of 10-20 strategies at that level can have a significant effect on which companies appear at which rank.
 
The second key area I found was that the data between the two systems is slightly different. Both Thomson Reuters (who I think P123 use) and Factset (our providers) normalize certain data points to their own definitions – this includes often used numbers such as EPS and Cash flows. I have gone through this with Factset a number of times and what they do makes sense, as I’m sure Reuters do – they are normalizing the data for us so that we can compare companies without some of the “financial games” that CFOs like to play getting in the way. This again makes comparing the same system on the two platforms very problematic (I had about 10 that I liked on P123, and only a couple were even partly successful on IE). Our view is that as long as these changes are applied consistently then we’re happy to use them. As an interesting note, we found that Factset were generally a lot closer to the SEC submissions than Reuters were, especially with smaller companies.”

In resume: I am not comparing identical processes to filtering the stocks and therefor will not get similar results.

I understand the point, but if we are talking about the real world, I think the same system must retorn the same results in the backtester, if not... how can I trust in that system? What is the better way to proceed? Should I trust in the sim one or in the sim two? Where should I place my money?

Well, your turn.: What do you think about that?

I attached few screen captures from the original system from the web x and what I translated to P123.

Attachment Captura de pantalla 2019-02-13 a las 18.44.47.png (132939 bytes) (Download count: 199)


Attachment Captura de pantalla 2019-02-13 a las 18.46.21.png (250477 bytes) (Download count: 200)


Attachment 30% anual liam copia.jpg (758293 bytes) (Download count: 199)


Mar 1, 2019 11:09:42 AM       
wwasilev
Re: Open question about different simulation platforms

There could be many explanations for the differences.

For starters, IE's universe definition looks incomplete. Are there minimum market cap and liquidity restrictions? Since P123 supports rules in Universe creation, try adding what the other site is assuming and get your universe count down from 4500 to about 922.

Another issue may be slippage. Many sims look great with 0% slippage and terrible with P123's variable slippage. It's critical to include slippage in the modeling. If the other site isn't doing that, ask them why.

And ignore the data provider issue. P123 uses an industry quality provider - S&P Global. We're wearing big boy pants now.

Walter

PS - I'll try recreating your sim this weekend.

PPS - It looks like they also use the following in their universe rules;

and have EPS(ltm) > EPS(a, 2)
and have NCAV > 0

Add the equivalent to your P123 Universe rules.

PPPS
1) for consecutive dividend increases, drop the equal to 10 comparison; since you want an actual count, use
LoopSum("DivPS(CTR,ANN)>=DivPS(CTR+1,ANN)",10)

2) P123 Yield is forward yield. For trailing yield, use something like
DivPaidTTM/MktCap


Mar 1, 2019 12:44:50 PM       
Edit 8 times, last edit by wwasilev at Mar 1, 2019 3:38:58 PM
mgerstein
Re: Open question about different simulation platforms

It’s always hard to compare platforms for reasons others have stated. I would point out though that the S&P-Compustat database we use is the platinum standard, not only in terms of standardization protocols (see white paper in Help section) but also in that what we use is true point in time, which FactSet does not have.

Just a side note on your model. All Stocks willl, indeed, give you many extremely low quality stocks. Many of our users choose All Fundamentals USA by default and others use PRussell3000. Forget nonsense like 30% annual alpha. That does not exit in the real world. Any alpha above zero is pretty good,

Your model refers in one place to EPSXclXor(2,ANN). In the context in which I think yu’re using it ad given our taxonomy, are you sure (2, ANN) is really what you want. Or is it (0,ANN), the last completed annual period which at times, may largely overlap TTM, or it is (1,ANN), the next period?

Are you copying the strategy definition from platform X? Did they explain why they want yield to be as high as possible? That will point you to companies with the worst near-term prospects, the ones most likely to cut or eliminate dividends. If I use a higher-is-better yield sort, I’m going to accompany it with a lot of rules designed to weed out companies at risk of dividend cuts, and I don’t see that in the strategy you’re replicating.

There’s also a bug in eps(ltm)/eps(ltm,1) - their terminology — and your adaptation. If one or both of the EPS figures is negtive, you’ll find a mess on your hands. You should use the format (a - b)/abs(b), or (eps(ltm) - eps(ltm,1))/abs(eps(ltm,1).

I understand your dilemna in wondering which platform is trustworthy. I know that being with portfolio123 I’m obviously biased in our favor, but for what it’s worth, might I suggest you go through our Help material and on-line courses and consider choosing the platform that works hard to support and empower you to create strategies that really make sense and offer a better probability of success in real life.

Marc Gerstein
Portfolio123, Director of Research
Twitter: @MHGerstein
Blog: https://actiquant.com/category/home/
Want to know what's going to happen in the next year? No problem; I'll tell you 366 days from now.

Mar 1, 2019 4:51:19 PM       
Edit 3 times, last edit by mgerstein at Mar 2, 2019 11:18:50 AM
yuvaltaylor
Re: Open question about different simulation platforms

I'd like to add that putting your buy rules in the universe rather than the simulation will definitely give you better results. That's due to the nature of ranking systems. If you put your buy rules in the universe, then ranking takes priority over the buy rules; if you put them in the simulation, then the buy rules take priority over the ranking. Because buy rules are so inflexible and ranking is so flexible, it makes more sense to put the rules in the universe whenever you can. It makes a real difference.

Yuval Taylor
Product Manager, Portfolio123
invest(igations)

Mar 1, 2019 6:57:48 PM       
jpeasy
Re: Open question about different simulation platforms

I'd like to add that putting your buy rules in the universe rather than the simulation will definitely give you better results. That's due to the nature of ranking systems. If you put your buy rules in the universe, then ranking takes priority over the buy rules; if you put them in the simulation, then the buy rules take priority over the ranking. Because buy rules are so inflexible and ranking is so flexible, it makes more sense to put the rules in the universe whenever you can. It makes a real difference.



Can you explain this a bit more? How does ranking take preference if you filter the universe by buy rules? Seems like the universe filter takes the largest preference by limiting the stocks the ranking system can see.

Mar 1, 2019 7:21:58 PM       
Jrinne
Re: Open question about different simulation platforms

It’s always hard to compare platforms for reasons others have stated. I would point out though that the S&P-Compustat database we use is the platinum standard, not only in terms of standardization protocols (see white paper in Help section) but also in that what we use is true pint in time, which FactSet does not have.


And the methods of developing a system are awesome!!!! By methods, I mean the rank performance, backtests, sims/ports. etc.

This includes the option of putting the rules in the universe or the sim as you see fit. P123 is guaranteed to have the better way (which ever one you decide is better).

The advice above is true, I think. But it can be made simpler: NEVER USE THE ALL STOCK UNIVERSE WITHOUT MODIFYING IT.

-Jim

From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Mar 2, 2019 7:16:16 AM       
Edit 12 times, last edit by Jrinne at Mar 2, 2019 10:18:37 AM
ivillalongabarreiro
Re: Open question about different simulation platforms

.

Mar 3, 2019 11:35:54 AM       
Edit 1 times, last edit by ivillalongabarreiro at Mar 3, 2019 11:37:22 AM
Monday33
Re: Open question about different simulation platforms

Hello Walter.

I tried what you said in PPPS 😂, and do not work, the sim goes to ruin anyway.

And the rules you refer in PPS are already in my sim.

Thanks a lot for your answer.

Mar 3, 2019 11:39:19 AM       
Monday33
Re: Open question about different simulation platforms

Hello Mgerstein.

I corrected what you said about EPSXclXor, yes I wanted to compare with the previous year, then (1,ANN) is the correct instruction.

But when you say “You should use the format (a - b)/abs(b), or (eps(ltm) – eps(ltm,1))/abs(eps(ltm,1).”

Do you refer at the formulas in my ranking system? What is “b”?

Is the correct formula like this?

(EPSExclXorLTM- EPSExclXorLTM,1) / abs(EPSExclXorLTM,1)

To be honest I feel P123 is a lot better than the other (X) platform, but a friend ask me to check that platform and it's just what I did.

And, by the way, I'm reading all the stuff you hang here, but since I'm not a matematician, nor an accountant, nor an informatic, and even nor a native english speaker... well, all this stuff it's complex to me.

Thanks a lot for your answer. :)

Mar 3, 2019 11:41:31 AM       
Monday33
Re: Open question about different simulation platforms

Hello Jim.

I do not know how to modifying the universe of stocks, except, of course, excluding some stocks at the end of the sim process in the list for the tickers.

Is there any other way?

And why should I want to do this?

Thanks!!!

Mar 3, 2019 11:45:50 AM       
Posts: 17    Pages: 2    1 2 Next
 Last Post