Open question about different simulation platforms

Hi all!!

I would like to comment something with you and get some feedback because I’m not sure about what to think.

Recently I tried another plattaform, very similar to Portofio 123, but little bit simplier. The staff from the plattaform send some systems to the users. One I recibed shows a very good performance, around 30% a year, then I translated that system to the Portfolio123 just to check if the results are the same, and… surprise, the sim goes to ruin in just few years.

*I’m not going to write down here the name of that platform, if anyone is interested ask me by private email and I will send you the adress.

I asked to the guys who run the plattaform about that difference in results. And they answer the following:

“The key difference I found where the two systems are different is that P123 generally takes a standard universe of all stocks, ranks it and then applies your buy rules, whereas “X” applies the universe rules, then ranks, then applies any buy rules. This is a much bigger difference than it sounds, as ranks act very differently with different numbers of securities in a universe. If you look at your Value strategy and click the Rank Screener option you can see that it includes just 922 securities in your list, whereas P123 will be ranking on approximately 4,500. Even a difference of 10-20 strategies at that level can have a significant effect on which companies appear at which rank.

The second key area I found was that the data between the two systems is slightly different. Both Thomson Reuters (who I think P123 use) and Factset (our providers) normalize certain data points to their own definitions – this includes often used numbers such as EPS and Cash flows. I have gone through this with Factset a number of times and what they do makes sense, as I’m sure Reuters do – they are normalizing the data for us so that we can compare companies without some of the “financial games” that CFOs like to play getting in the way. This again makes comparing the same system on the two platforms very problematic (I had about 10 that I liked on P123, and only a couple were even partly successful on IE). Our view is that as long as these changes are applied consistently then we’re happy to use them. As an interesting note, we found that Factset were generally a lot closer to the SEC submissions than Reuters were, especially with smaller companies.”

In resume: I am not comparing identical processes to filtering the stocks and therefor will not get similar results.

I understand the point, but if we are talking about the real world, I think the same system must retorn the same results in the backtester, if not… how can I trust in that system? What is the better way to proceed? Should I trust in the sim one or in the sim two? Where should I place my money?

Well, your turn.: What do you think about that?

I attached few screen captures from the original system from the web x and what I translated to P123.




There could be many explanations for the differences.

For starters, IE’s universe definition looks incomplete. Are there minimum market cap and liquidity restrictions? Since P123 supports rules in Universe creation, try adding what the other site is assuming and get your universe count down from 4500 to about 922.

Another issue may be slippage. Many sims look great with 0% slippage and terrible with P123’s variable slippage. It’s critical to include slippage in the modeling. If the other site isn’t doing that, ask them why.

And ignore the data provider issue. P123 uses an industry quality provider - S&P Global. We’re wearing big boy pants now.

Walter

PS - I’ll try recreating your sim this weekend.

PPS - It looks like they also use the following in their universe rules;

and have EPS(ltm) > EPS(a, 2)
and have NCAV > 0

Add the equivalent to your P123 Universe rules.

PPPS

  1. for consecutive dividend increases, drop the equal to 10 comparison; since you want an actual count, use LoopSum("DivPS(CTR,ANN)>=DivPS(CTR+1,ANN)",10)
  2. P123 Yield is forward yield. For trailing yield, use something like DivPaidTTM/MktCap

It’s always hard to compare platforms for reasons others have stated. I would point out though that the S&P-Compustat database we use is the platinum standard, not only in terms of standardization protocols (see white paper in Help section) but also in that what we use is true point in time, which FactSet does not have.

Just a side note on your model. All Stocks willl, indeed, give you many extremely low quality stocks. Many of our users choose All Fundamentals USA by default and others use PRussell3000. Forget nonsense like 30% annual alpha. That does not exit in the real world. Any alpha above zero is pretty good,

Your model refers in one place to EPSXclXor(2,ANN). In the context in which I think yu’re using it ad given our taxonomy, are you sure (2, ANN) is really what you want. Or is it (0,ANN), the last completed annual period which at times, may largely overlap TTM, or it is (1,ANN), the next period?

Are you copying the strategy definition from platform X? Did they explain why they want yield to be as high as possible? That will point you to companies with the worst near-term prospects, the ones most likely to cut or eliminate dividends. If I use a higher-is-better yield sort, I’m going to accompany it with a lot of rules designed to weed out companies at risk of dividend cuts, and I don’t see that in the strategy you’re replicating.

There’s also a bug in eps(ltm)/eps(ltm,1) - their terminology — and your adaptation. If one or both of the EPS figures is negtive, you’ll find a mess on your hands. You should use the format (a - b)/abs(b), or (eps(ltm) - eps(ltm,1))/abs(eps(ltm,1).

I understand your dilemna in wondering which platform is trustworthy. I know that being with portfolio123 I’m obviously biased in our favor, but for what it’s worth, might I suggest you go through our Help material and on-line courses and consider choosing the platform that works hard to support and empower you to create strategies that really make sense and offer a better probability of success in real life.

I’d like to add that putting your buy rules in the universe rather than the simulation will definitely give you better results. That’s due to the nature of ranking systems. If you put your buy rules in the universe, then ranking takes priority over the buy rules; if you put them in the simulation, then the buy rules take priority over the ranking. Because buy rules are so inflexible and ranking is so flexible, it makes more sense to put the rules in the universe whenever you can. It makes a real difference.

Can you explain this a bit more? How does ranking take preference if you filter the universe by buy rules? Seems like the universe filter takes the largest preference by limiting the stocks the ranking system can see.

And the methods of developing a system are awesome!!! By methods, I mean the rank performance, backtests, sims/ports. etc.

This includes the option of putting the rules in the universe or the sim as you see fit. P123 is guaranteed to have the better way (which ever one you decide is better).

The advice above is true, I think. But it can be made simpler: NEVER USE THE ALL STOCK UNIVERSE WITHOUT MODIFYING IT.

-Jim

.

Hello Walter.

I tried what you said in PPPS :joy:, and do not work, the sim goes to ruin anyway.

And the rules you refer in PPS are already in my sim.

Thanks a lot for your answer.

Hello Mgerstein.

I corrected what you said about EPSXclXor, yes I wanted to compare with the previous year, then (1,ANN) is the correct instruction.

But when you say “You should use the format (a - b)/abs(b), or (eps(ltm) – eps(ltm,1))/abs(eps(ltm,1).”

Do you refer at the formulas in my ranking system? What is “b”?

Is the correct formula like this?

(EPSExclXorLTM- EPSExclXorLTM,1) / abs(EPSExclXorLTM,1)

To be honest I feel P123 is a lot better than the other (X) platform, but a friend ask me to check that platform and it’s just what I did.

And, by the way, I’m reading all the stuff you hang here, but since I’m not a matematician, nor an accountant, nor an informatic, and even nor a native english speaker… well, all this stuff it’s complex to me.

Thanks a lot for your answer. :slight_smile:

Hello Jim.

I do not know how to modifying the universe of stocks, except, of course, excluding some stocks at the end of the sim process in the list for the tickers.

Is there any other way?

And why should I want to do this?

Thanks!!!

The PPPS rules worked for me when there are liquidity constraints added to the universe. Make your simulation public if you want more feedback.
And are the PPS rules in your buy rules or universe rules?

Walter

PS I used LoopSum(“DivPS(CTR,ANN)>=DivPS(CTR+1,ANN)”,10) to replicate IE’s ConsecutiveIncreasesOrEqual(Dividends(a), 10). That may be insufficient. I don’t know.

PPS This should be better; ShowVar(@ConsecutiveOrEqualDivpaid,LoopSum("Eval(DivPaid(CTR,ANN)>=DivPaid(CTR+1,ANN),1,NA)",10,0,1,0,1)) Since this works even when DivPaidA=0, make sure only dividend paying stocks are considered or add a test for DivPaidA=0. BTW, IE’s ConsecutiveIncreasesOrEqual function looks broken to me. Oh well.

Yes. That formula would be used any time you’re computing a growth rates and one of the two numbers might be negative.

Neither am I.

You’re not alone. There are other experienced members who can say the same thing.

Just be patient. Learn at your own pace. And when you have questions, ask (as you did here). As you can see, there are many members here who are happy to help.

Hello Walter.

Sorry for my late answer, I’ve been busy these days.

The sim is public, I posted all the information I have in my first post in this thread, I have nothing else.

And to be honest I do not know where are my PPS rules… I think they are in both of them, in the buy rules and also in the Universe rules ( I think you mean Ranking rules here… )

The ShowVar instruction I think do not work inside the ranking Ranking system formulas, or when I placed this code (ShowVar(@ConsecutiveOrEqualDivpaid,LoopSum(“Eval(DivPaid(CTR,ANN)>=DivPaid(CTR+1,ANN),1,NA)”,10,0,1,0,1)))

the program says there is an error near ShowVar.


Hello Marc

Thanks for your answer, and for cheering me up. I’ll try to do my best. :slight_smile:

By the way the formula you told me, (EPSExclXorLTM- EPSExclXorLTM,1) / abs(EPSExclXorLTM,1), seems not to work inside the Ranking system. I do not know why…

The program says that there is an error near EPSExclXorLTM…


ShowVar only works in the Screener. Sorry for not being clear about that.

As for the original issue about differences in simulation results, while the other site looks like a P123 clone, there are enough gaps in my understanding of their implementation that it is unclear where the differences originate - it could be data provider issues, or how NAs are handled in the ranker, etc.

Also, I looked at a few of their example model and I would not use them myself.

Walter

Yes yes, I’m agree with you. Maybe I’ll translate some other of their systems to P123 but I do not trust so much on it…

I’m wondering if that syntax is for a different system? Try this syntax for P123 EPS growth proxy:

(EPSExclXorTTM- EPSExclXorPTM) / abs(EPSExclXorPTM)